International Journal of
ISSN 0144-3577
Operations & Production Management
Volume 22 Number 11 2002
Performance measurement Guest Editors Umit Bititci and Jill MacBryde Paper format International Journal of Operations & Production Management includes 12 issues in traditional paper format. The contents of this issue are detailed below.
Internet Online Publishing with Archive, Active Reference Linking, Emerald WIRE, Key Readings, Institution-wide Licence, E-mail Alerting Service and Usage Statistics. Access via the Emerald Web site: http://www.emeraldinsight.com/ft See overleaf for full details of subscriber entitlements.
Access to International Journal of Operations & Production Management online ________________________ 1190 Editorial board ________________________________________ 1191 Abstracts and keywords _______________________________ 1192 About the Guest Editors _______________________________ 1194 Guest editorial_________________________________________ 1196 Modelling design development performance F.J. O’Donnell and A.H.B. Duffy __________________________________ 1198
A framework of the factors affecting the evolution of performance measurement systems Mike Kennerley and Andy Neely __________________________________ 1222
Adding value to performance measurement by using system dynamics and multicriteria analysis Se´rgio P. Santos, Valerie Belton and Susan Howick___________________ 1246
Web enabled performance measurement systems: management implications U.S. Bititci, S.S. Nudurupati, T.J. Turner and S. Creighton ____________ 1273
The success and failure of performance measurement initiatives: perceptions of participating managers Mike Bourne, Andy Neely, Ken Platts and John Mills _________________ 1288
This issue is part of a comprehensive multiple access information service
CONTENTS
IJOPM 22,11
International Journal of Operations & Production Management online An advanced knowledge resource for the entire organization Access via the Emerald Web site – http://www.emeraldinsight.com/ft
1190
Subscribers to this journal benefit from access to a fully searchable knowledge resource stretching far beyond the current volume and issue. International Journal of Operations & Production Management online is enhanced with a wealth of features to meet the need for fast, effortless, and instant access to the core body of knowledge. Furthermore, this user-friendly electronic library may be networked throughout the subscribing organization to maximize the use and value of the subscription. This is augmented with advanced search facilities and ‘‘choice of access’’ through a variety of alternative journal gateways.
Emerald online access includes: Institution-wide Licence Our liberal licence allows everyone within your institution to access the journals via the Internet, making your subscription more cost-effective. It has been designed to provide a comprehensive, simple system with minimum administration. Access can be granted by IP range or username and password. ATHENS authentication is enabled.
Current Issue + Journal Archive Internet access via Emerald Fulltext to information as it becomes available online and to material from past volumes. Users can choose to browse the database for relevant articles or search using the powerful search engine provided.
Active Reference Linking Direct links from the journal article references to abstracts of the most influential articles cited. Where possible, this link is to the full text of the article.
Emerald WIRE A fully searchable subject specific database, brought to you by Emerald Management Reviews, providing article abstracts from the world’s top management, computing and engineering journals. Directly relating to the journal subject area, the database is updated monthly and gives users details of how to obtain the full text of original articles.
interesting articles from other publications in the field. The abstracts are available online and are updated regularly upon publication of the latest issue.
Emerald Alert The table of contents e-mail alerting service will e-mail you the contents page of any chosen journal whenever the latest issue becomes available online. For further information please go to http://www.emeraldinsight. com/alerts
Support Resources A comprehensive range of resources is available online that help users learn how to effectively use online information resources and that helps information professionals market resources to their users. For further information please go to http://www.emeraldinsight. com/support
Choice of Access Electronic access to this journal is available via a number of channels, enabling end users’ libraries to reach the content through their preferred delivery system. The Emerald Web site – http://www. emeraldinsight. com/ft – is the recommended means of electronic access as it provides fully searchable and high value-added access to the complete content of the journal. Refer to the next section for ‘‘how to access’’ via the Emerald Web site. Subscribers can also access and search the article content of this journal through the following journal gateway services: Ebsco Online http://www.ebsco.com/online Huber E-Journals http://e-journals.hanshuber.com/english Minerva http://www.minerva.at/ OCLC Firstsearch ‘‘Electronic Collections Online’’ http://www.uk.oclc.org/oclc/menu/eco.htm RoweCom’s ‘‘Information Quest’’ http://www.informationquest.com SilverPlatter http://www.silverplatter.com SwetsBlackwell’s ‘‘SwetsnetNavigator’’ http://www.swetsnetnavigator.nl
Usage Statistics Online Journal Usage Statistics are now available. This feature allows Emerald Administrators to download their usage statistics with regard to their organization’s journal usage. Usage Statistics allow you to review the value of electronic dissemination of your journal subscriptions throughout your organization. They can also help determine the future trends for information within your organization. For further information go to http://www.emeraldinsight.com/stats
Key Readings Abstracts of articles, relating to keywords, are selected to provide readers with current awareness of
How to access this journal through Emerald Organizations must first register for online access (instructions provided at http://www.emeraldinsight. com/register), after which the content is available to everyone within the organization’s domain. To access this journal’s content, simply log on either from the journal homepage or direct through the Emerald Web site. Emerald Customer Support Services For customer service and technical help, contact: E-mail:
[email protected] Telephone: (44) 1274 785278 Fax: (44) 1274 785204
Editorial board
EDITORIAL BOARD Dr Linda Angell Victoria University of Wellington, New Zealand Professor David Bennett (President: European Operations Management Association) Aston Business School, Birmingham, England Professor Will Bertrand Eindhoven University of Technology, The Netherlands Dr Ruth Boaden Manchester School of Management, UMIST, Manchester, England Professor Richard B. Chase School of Business Administration, University of Southern California, USA Professor T.C. Edwin Cheng Hong Kong Polytechnic University, Hong Kong Professor Henrique Luiz Correa Fundacao Getulio Vargas-Eaesp, Avenida Nove de Julho 2029-10, Andar 01313-902, Sao Paulo SP, Brazil Professor Roberto Filippini Universita´ di Padova, Vicenza, Italy Professor Christer Karlsson IMIT, Stockholm School of Economics, Stockholm, Sweden Professor Hiroshi Katayama Department of Industrial Engineering, Waseda University, Tokyo, Japan
Professor Douglas Macbeth Management Studies, Glasgow University Business School, Glasgow, Scotland Professor Jack Meredith Wake Forest University, USA Professor Alan Muhlemann University of Bradford Management Centre, Bradford, England
1191
Professor Colin New School of Management, Cranfield University, Cranfield, England Professor Abdul Raouf Institute of Leadership and Management, Lahore, Pakistan Professor Nigel Slack Warwick Business School, Coventry, England Professor Amrik S. Sohal Department of Business Management, Monash University, Caulfield East, Victoria, Australia Professor Chris Voss London Business School, England Professor Prem Vrat Director, Indian Institute of Technology, Roorkee, India Professor Mohamed A. Youssef Department of Decision Sciences, Norfolk State University, USA
International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, p. 1191. # MCB UP Limited, 0144-3577
IJOPM 22,11
1192
International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, Abstracts and keywords. # MCB UP Limited, 0144-3577
Modelling design development performance F.J. O’Donnell and A.H.B. Duffy Keywords Performance measurement, Modelling, Efficiency Begins with a brief review of research in the area of performance, with particular emphasis on design and product development, highlighting the main weaknesses in work to date. Then presents a fundamental and generic model of performance, related to knowledgebased activities in design, which describes performance in terms of its key elements, efficiency and effectiveness, and provides a basis for modelling performance across different process levels. Evolves this model further to describe performance measurement and management in design development. Concludes that it is not possible to distinguish the efficiency of the design activity from that of design management, and that, in addition, efficiency appears to be inherent to an activity, but is difficult to measure. A framework of the factors affecting the evolution of performance measurement systems Mike Kennerley and Andy Neely Keywords Performance measurement, Development, Organizational change The effectiveness of performance measurement is an issue of growing importance to industrialists and academics alike. Many organisations are investing considerable amounts of resource implementing measures that reflect all dimensions of their performance. Consideration is being given to what should be measured today, but little attention is being paid to the question of what should be measured tomorrow. Measurement systems should be dynamic. They have to be modified as circumstances change. Yet few organisations appear to have systematic processes in place for managing the evolution of their measurement systems and few researchers appear to have explored the question, what shapes the evolution of an organisation’s measurement system? The research reported in this paper seeks to address this gap in the literature by presenting data that describes the forces that shape the evolution of the measurement systems used by different organisations.
Adding value to performance measurement by using system dynamics and multicriteria analysis Se´rgio P. Santos, Valerie Belton and Susan Howick Keywords Performance measurement, Management, Systems design, Integration The design, implementation and use of adequate performance measurement and management frameworks can play an important role if organisations are to succeed in an increasingly complex, interdependent and changing world. Yet, despite widespread recognition of the importance of performance assessment, there are some issues that require further study if measurement systems are to be effective in the process of management through monitoring and decision making. This article proposes and seeks to illustrate that the use of system dynamics and multicriteria decision analysis, as part of a broader framework for performance measurement, can address some of the issues identified. The use of these approaches, independently and in an integrated manner, provides a means of exploring the dynamic complexity of organisations, making explicit trade-offs between performance measures, and assessing the impacts of initiatives to improve performance, thereby enhancing understanding and ultimately contributing to the improvement of organisational performance. Web enabled performance measurement systems: management implications U.S. Bititci, S.S. Nudurupati, T.J. Turner and S. Creighton Keywords Performance measurement, Business culture, Information technology, Flexibility, Internet In order to respond proactively to the new emerging challenges, management requires upto-date and accurate information on performance. Such performance measurement systems are still not common because there is not sufficient research focused on management implications of IT enabled performance measurement systems. This paper presents an empirical case study where a fully integrated IT enabled performance
measurement system was implemented and its management implications studied. The paper concludes, based on a single case study, that if properly implemented, such systems would promote a proactive management style and greater confidence in management decisions. The success and failure of performance measurement initiatives: perceptions of participating managers Mike Bourne, Andy Neely, Ken Platts and John Mills Keywords Performance measurement, Management, Implementation This paper investigates the success and failure of performance measurement system design interventions in ten companies. In each
case, the senior management team was facilitated through a management process to redesign their performance measurement systems. Analysis of the initial pilot case studies suggested three important differences between companies that proceeded to implement the measures agreed during the process and those that did not. Post intervention semi-structured interviews with the directors and managers directly involved revealed two main perceived drivers of implementation and four perceived factors that block implementation. The results are of specific interest for performance measurement system implementation but have wider implications for our view of management commitment in change management.
Abstracts and keywords
1193
IJOPM 22,11
1194
International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, pp. 1194-1195. # MCB UP Limited, 0144-3577
About the Guest Editors Umit Bititci Umit Bititci is the Founder and Director of the Centre for Strategic Manufacturing (CSM), which is based in the University of Strathclyde, Glasgow, UK. CSM is a centre of excellence in research, development and technology transfer. The Centre works closely with industry, assisting manufacturing companies to improve their performance through development, adoption and implementation of new concepts, practices techniques and technologies. Since its foundation, the Centre has developed an impressive list of business partners and clients from various parts of both the UK and Europe. Umit has 14 years of industrial, consultancy and academic experience in the field of manufacturing engineering, operations and strategic management. In his current position he also acts as a board level technology and development advisor with Highland Spring Ltd and Daks Simpson Ltd. Umit has gained international and national recognition for his work on performance measurement from both the academic and industrial communities. His contribution to this field is reflected through more than 90 publications. He is regularly invited to talk at national and international seminars and conferences. He is also regularly interviewed and consulted by press and media for his expertise in strategic management in manufacturing industries. As an academic, Umit has secured over £2 million R&D funding in 18 research and development contracts. He has been responsible for development and success of the department’s masters programme, management of competitive manufacturing, on which he teaches strategy and technology management. He has been invited to lecture as a visiting professor at various European institutions on strategy, performance and technology management. As a consultant, he has worked with a wide range of multinational blue chip and SME clients, assisting and facilitating strategic change through implementation of new technologies and practices. Umit’s consultancy career started during the mid-1980s and since then he has worked with over 30 companies from various industries, including FMCG, and food and drink companies, such as Grand Metropolitan (IDV), Cinzano, Telford Foods, R.F. Brooks and Highland Spring. In his previous position as a senior consultant with Strathclyde Institute he was also responsible for managing and co-ordinating the activities of a team of consultants. Jill MacBryde Jill MacBryde is the Deputy Director of the Centre for Strategic Manufacturing, Glasgow, UK. She also holds the post of Director of Teaching and Learning within the Department of Design, Manufacture and Engineering Management, at the University of Strathclyde.
Jill’s research focuses on process improvement. Jill is currently the principal About the Guest investigator on a research project focusing on measuring and managing Editors employee and team performance in business process environments. This work has been funded by the Economic & Physical Sciences Research Council (EPSRC). Jill is a graduate of the University of Strathclyde where she gained a degree 1195 in technology and business studies. Following graduation she joined McCall Information Systems, first, in a business development role latterly in a project management role. She then worked with the Emerging Technologies Research & Assessment Centre within the Strathclyde Business School. In this role Jill carried out audits within a number of universities identifying exploitable technologies and developing commercial plans. She then joined the Department of Design, Manufacture & Engineering Management as a Lecturer in engineering management. Since joining the department in 1994 Jill’s research and teaching has focused on business process improvement and performance measurement. She has also been actively involved in a number of project management and consultancy assignments, directly relating to process improvement and performance management. These assignments have been for major blue chip companies as well as SMEs. Throughout her career, Jill has worked with a wide spectrum of clients on assignments such as: business process redesign; performance measurement; process improvement; team performance; and managing change. Jill has conducted consultancy and training assignments with clients in the UK, Europe and Asia and Australia.
IJOPM 22,11
1196
International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, pp. 1196-1197. # MCB UP Limited, 0144-3577 DOI 10.1108/01443570210450275
Guest editorial This special issue is based on selected papers presented at the 1st International Performance Measurement Workshop organised by the Performance Measurement Special Interest Group of the IFIP WG 5.7. The workshop took place on 24-25 May 2001. IFIP (www.ifip.or.at) is a non-governmental, non-profit umbrella organisation for national societies working in the field of information processing. It was established in 1960 under the auspices of UNESCO as an aftermath of the first World Computer Congress held in Paris in 1959. Today, IFIP has several types of members and maintains friendly connections to specialized agencies of the UN system and non-governmental organizations. Technical work, which is the heart of IFIP’s activity, is managed by a series of Technical Committees. WG5.7 is the Working Group 7 of IFIP’s Technical Committee No. 5. The aim of WG 5.7 is to promote and encourage the advancement of knowledge and practice in the field of integrated production and operations management to maximise global dissemination of this knowledge (www.ecopro.polimi.it/ifipwg5-7). This broad aim is achieved by: . continuous development and refinement of a research agenda; . developing a research culture that nurtures research that addresses industrial need whilst maintaining academic excellence; and . disseminating R&D results and best practices globally to both academics and practitioners through the groups’ annual conference and the activities of its special interest groups. The Special Interest Group (SIG) on performance measurement was established in 2000 at the committee’s annual meeting which was held in Berlin. Since then its membership has grown to include 76 individuals from 24 countries. Following a number of initial plenary meetings, the group held its 1st International Workshop on 24-25 May 2001 in Glasgow, Scotland. The second workshop was held in 6-7 June 2002 in Hanover, Germany and the third workshop is scheduled to take place to in Italy to coincide with the EurOMA/ POMS conference and the 1st meeting of the recently established Networks of Excellence under the Auspices of the international Intelligent Manufacturing Systems (IMS) programme. The workshops organised by the SIG are unique in their character. The workshop encourages submission of working papers where a discusser is appointed for each paper and ample time is allocated to facilitate an in-depth, critical but constructive discussion. The objective is to provide feedback to the authors so that they can develop their research and produce journal quality papers. The papers presented in this special issue are based on the papers presented at the SIGs first workshop which took place on 24-25 May 2001 in Glasgow.
Since then, the papers have been edited and upgraded by the authors based on the feedback they received following the workshop. Finally, all the papers have been subjected to IJOPM’s rigorous double blind refereeing process in order to qualify for publication in this special issue. The papers included in this special issue provide different perspectives to performance measurement and management. The paper by F.J. O’Donnell and A.H.B. Duffy looks at how we measure the performance of design activities. In doing this they take us back to basics and introduce a tool for modelling the performance of the design development process. We felt that this formalism, which splits the design development process into two types of activities – i.e. design management activities and design activities, can be used with similar rigour in modelling processes other than design development. The paper by Mike Kennerley and Andy Neely provides a different perspective by demonstrating the dynamic nature of organisations and presenting us with evidence of how few companies have, in place, a process for managing the evolution of their performance measurement systems. Their paper also highlights how little research has been done in this area. Se´rgio Santos, Valerie Belton and Susan Howick develop this theme of the dynamic nature of organisations in their paper. They too identify some key issues requiring further research if performance measurement systems are to be fully utilised in a dynamic fashion. They consider system dynamics and multicriteria decision analysis as having potential to deal with the complexity and change in organisations – providing a way of exploring the dynamic complexity and looking at trade-offs between performance measures in changing environments. The paper by Umit Bititci (one of the editors), S.S. Nudurupati, T.J. Turner and S. Creighton, whilst continuing the theme of dynamic, proactive performance measurement systems, also introduces the notion that provision of a properly designed and interactive performance measurement system, which operates in near-real-time, could have a positive impact on the organisational behaviour and management style, thus acting as an aid to management of change. Finally, the paper by Mike Bourne, Andy Neely, Ken Platts and John Mills, in researching the reasons for success and failure of performance measurement projects, further develops the organisational development and management of change perspective to performance measurement and management. As joint editors we are very pleased to be able to present in this special issue five complementary papers that provide a rich insight into the current research in this important area. We would like to extend our gratitude to all the authors, discussers and referees who made this issue possible. In compiling this issue we certainly gained tremendous satisfaction and enjoyment. We hope that you will also find these papers equally stimulating and informative. Umit Bititci Jill MacBryde
Guest editorial
1197
The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/0144-3577.htm
IJOPM 22,11
Modelling design development performance F.J. O’Donnell
1198
Scottish Enterprise, Glasgow, UK, and
A.H.B. Duffy CAD Centre, DMEM, University of Strathclyde, Glasgow, UK Keywords Performance measurement, Modelling, Efficiency Abstract Begins with a brief review of research in the area of performance, with particular emphasis on design and product development, highlighting the main weaknesses in work to date. Then presents a fundamental and generic model of performance, related to knowledge-based activities in design, which describes performance in terms of its key elements, efficiency and effectiveness, and provides a basis for modelling performance across different process levels. Evolves this model further to describe performance measurement and management in design development. Concludes that it is not possible to distinguish the efficiency of the design activity from that of design management, and that, in addition, efficiency appears to be inherent to an activity, but is difficult to measure.
International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, pp. 1198-1221. # MCB UP Limited, 0144-3577 DOI 10.1108/01443570210450301
1. Introduction The overriding aim of much of the engineering design research is to improve the performance of the design process, and consequently the product development process. Much has been written within the product development literature on the performance of the product development process (Brown and Eisenhart, 1995). This work has been largely focused on the analysis of performance at the project or program level. The ability to relate the different research and draw generic lessons from the results has been stifled by the lack of consistency on the meaning of performance both at a generic level (Neely, 1999) and more specifically in design development (Montoya-Weiss and Calantone, 1994). For example, although product and process performance have been distinguished within existing work we are unclear on how these relate or may be managed effectively. This paper begins with a brief review of research in the area of performance, with particular emphasis on design and product development, highlighting the main weaknesses in work to date. A fundamental and generic model of performance, related to knowledge-based activities in design, is then presented. The model describes performance in terms of its key elements, efficiency and effectiveness, and provides a basis for modelling performance across different process levels. This model is further evolved to describe performance measurement and management in design development. Design development is used here to indicate that both the design and its development are being considered when analysing performance.
2. Research in design performance Modelling design The research reviewed here forms part of the overall research in the area of development performance of organisations. Some of the work is generic in terms of being performance applicable across all business processes, while other work is aimed at more specific processes such as product development, design, and manufacturing (Figure 1)[1]. Within such areas the type of research and focus may vary widely 1199 and include empirical studies aimed at determining relationships between performance in different processes, the design and implementation of approaches for measuring performance, and the development of theoretical performance models. 2.1 Trends in performance research There has been considerable research published in the area of performance, e.g. Neely (1999) identified that between 1994 and 1996 some 3,615 articles on performance measurement were published. He refers to the growth in membership of accountancy institutes, and number of conferences on performance, as indicators of the increased interest in this area. However, in comparison to areas such as manufacturing, measuring the performance in product design is relatively undeveloped. For example, at the recent PM2000 conference in the UK there were no papers focused specifically on the analysis of design development performance from a list of over 90. Many authors have recognised the particular difficulties in measuring the performance in design development activities (Brookes and Backhouse, 1998; Chang and Yong, 1991; McGrath, 1994). These difficulties arise from the less tangible nature of outputs from design activities, such as being knowledge based, the often long duration and wide range of influences from design to market launch, or the difficulty in defining and measuring design quality. The decline of management accounting as the only way to measure business performance is an indication of the move towards measuring less tangible aspects of performance, e.g. those related to knowledge-intensive activities in design. Johnson and Kaplan (1987) suggest that traditional accounting methods are unsuited to organisations where the product life cycle is short and research and development assume increased importance.
Figure 1. Areas and types of performance related research
IJOPM 22,11
1200
Within the scope of this paper two areas of existing research in performance are briefly reviewed, i.e. the definition or modelling of performance and the relationship between design and design activity performance. 2.2 Defining and modelling performance The literature on performance is characterised by a lack of, and inconsistency in, definition of terms. Numerous works have been published that directly address the area of performance but do not explicitly define performance itself. Neely et al. (1995) in their review of performance literature suggest that ‘‘Performance measurement is a topic which is often discussed but rarely defined’’. Meyer and Gupta (1994) suggest that there is ‘‘massive disagreement as to what performance is’’ and that the proliferation of performance measures has led to the ‘‘paradox of performance’’, i.e. that ‘‘organisational control is maintained by not knowing exactly what performance is’’. That is, the lack of a comprehensive understanding of performance can often lead to ignorant acceptance of, for instance, particular approaches or metrics proposed by senior management in an organisation. A number of authors offer explicit definitions of performance, notably in Cordero (1989), Dwight (1999), Neely et al. (1996) Rolstadas (1998), and others refer to dimensions of performance (Clark and Fujimoto, 1991; Doz, 1996; Emmanuelides, 1993; Moseng and Bredrup, 1993). Table I presents a list of performance, and performance related, definitions highlighting the context in which the definitions were presented[2]. Some definitions of performance may be inferred from the information given, e.g. in van Drongelen and Cook (1997) and Sinclair and Zairi (1995) performance may be inferred to mean the attainment of objectives. Table I serves to illustrate the lack of consistency in defining performance within the literature, although authors may often be discussing the same phenomenon. However, some key terms emerge, i.e. efficiency and effectiveness are often cited in performance articles although not used specifically to define performance (Chang and Yong, 1991; Doz, 1996; Pritchard, 1990; Christainsen, 1993; Griffin, 1996; McDonough and Griffin, 1996). In general, effectiveness is related to the attainment of objectives or goals and efficiency is seen to relate to the use of resources. Although efficiency and effectiveness are used to describe performance, the relationship between these elements has not been defined. In some of the literature, metrics such as focus in development, adaptability and flexibility have been used. These metrics do not measure performance itself, but rather act as influences on it. For example, flexibility is only appropriate within an environment where changes are required and the capability for a process, such as design or manufacture, to change rapidly may add unnecessary costs or overheads in a stable environment. Flexibility will influence the performance, i.e. efficiency and/or effectiveness of an activity or process, but does not constitute a dimension of performance itself. In summary, the research in performance has been hindered by a lack of clarity on its meaning. In particular:
Total product quality, lead time and productivity (level of resources used) Focus in development, speed of development and R&D efficiency
Performance Performance Performance
Dimensions of performance Dimensions of performance Dimensions of performance
Dwight (1999)
Neely et al. (1996)
Rolstadas (1998)
Clark and Fujimoto (1991)
Doz (1996)
Emmanuelides (1993)
Time, cost, quality and flexibility
Neely et al. (1995)
Dimensions of performance
Efficiency, effectiveness and adaptability
Moseng and Bredrup (1993) Dimensions of performance
Development time, development productivity (use of resources) and total design quality
A complex inter-relationship between seven performance criteria: effectiveness efficiency quality productivity quality of work life innovation profitability/budgetability
Efficiency and effectiveness of purposeful action
The level to which a goal is attained
Effectiveness (i.e. measuring output to determine if they help accomplish objectives) Efficiency (i.e. measuring resources to determine whether minimum amounts are used in the production of these outputs)
Performance
Cordero (1989)
Definition
Element defined
Author and source
Manufacturing
Manufacturing (continued)
Product development (project)
Product development
Product development
Organisational system
Business
General
Research and development, organisation
Context
Modelling design development performance 1201
Table I. Performance and performance related definition
Performance measurement
Performance measurement
Efficiency Productivity Design productivity Design productivity
van Drongelen and Cook (1997)
Sinclair and Zairi (1995)
Andreasen and Hein (1987)
Griffin and Page (1993)
Duffy (1998)
Goldschmidt (1995)
Table I. Element defined
Efficiency and effectiveness
Efficiency and effectiveness
A measure of how well resources are combined and used to accomplish specific, desirable results
Ratio of increase in (clarification + risk reduction + detail + documentation) TO (increase in costs)
The process of determining how successful organisations or individuals have been in attaining their objectives
The acquisition and analysis of information about the actual attainment of company objectives and plans, and about factors that may influence this attainment
Definition
Engineering design
General
Product development
Organisations, individuals
General
Context
1202
Author and source
IJOPM 22,11
.
.
.
The key elements of performance have not been consistently defined or Modelling design agreed. development Those defining performance as efficiency and effectiveness have not distinguished them clearly or related them within a formalism of performance.
performance
Many of the measures used in the research relate to influences on performance and not performance itself.
1203
2.3 Design and design activity performance Design activity modelling has received significant attention over the years, aiming at the development of both descriptive and prescriptive models. This has resulted in the development of models offering different viewpoints of the design process such as the prescription of the activities and stages in design and their logical sequence. Others focused on the cognitive nature of design and those relating design within an overall model of product development. These models are aimed at increasing our understanding of design (descriptive), and providing a framework (such as procedures or guidelines) in which to carry out design (prescriptive), so that the level of performance may be maintained or improved. However, performance in design requires continued attention to both the design (artefact) and the activities involved in producing that design. That is, both design goals (DG) and design activity goals (DAG) exist within design development, and performance in relation to these goals must be distinguished yet related to overall performance. DGs relate to aspects of the design (artefact), such as its functionality, form, or behaviour, while DAGs relate to the activities in design development and consider aspects such as the time taken and cost of resources. Existing models of activities in design are almost exclusively focused on the performance of the design (artefact) and not the performance of the activities required for its development. For example: .
.
Radcliffe (1996) highlights the importance that designers place on design management activities within the design process in his protocol analysis but the analysis does not identify how design and design management activities are inter-related. Indeed, throughout the collection of papers from the delft workshop, analysing design activity (Cross et al., 1996), the analysis is restricted to the achievement of design (artefact) goals. The model of Pahl and Beitz (1996) is representative of a number of stage based models and provides a step-by-step method to be followed in design. However, the tasks outlined in this model are focused on DGs and there is no reference to activity goals and the need to manage the design process in relation to both design and activity goals, e.g. the trade-off between cost of design development and quality of the design.
IJOPM 22,11
1204
.
Authors such as Andreasen and Hein (1987) and Hales (1993) provide more insight into the (business) context in which design is carried out. Andreasen identifies the need for greater efficiency in product development while also ensuring better results in terms of the artefacts produced. The concept of efficiency as defined by Andreasen identifies the trade-off between what is being achieved in product development and the costs (and implicitly time) incurred. However, the author provides a viewpoint, identifying the need for managing such a tradeoff, and does not relate this within an activity or process model to further illustrate how it might be achieved.
The design activity and process models discussed provide significant insight into the activities and stages in design. There is a reasonable consensus on the main types of activities involved in design, their sequence, and the evaluation of the output in relation to the DGs is a key component of the models discussed. However, the analysis of performance in relation to the activities carried out in design is restricted to literature addressing the management of design at the project level (e.g. Hales, 1993). It is proposed here that management activities are carried out at every level in design and not just at a project level, and therefore there is a requirement to analyse performance in relation to activities at all levels. 3. A design performance model – E2 An activity model is presented here (Figure 2) focusing on knowledge in design. This model is based on IDEFØ (Colquhoun et al., 1993), one of the integrated computer aided manufacturing definition (IDEF) techniques, which was specifically created to model activities, processes or functions. IDEFØ supports the modelling of activities and their inter-relations; however, the focus of performance per se is on knowledge of those activities or processes. Further, IDEFØ does not explicitly represent the elements of performance. Design may be seen as the processing of knowledge (Persidis and Duffy, 1991), i.e. knowledge is continuously evolved as a result of specific activities between extremes of abstract versus concrete and
Figure 2. Knowledge processing activity
general versus specific (Andreasen, 1994; Ullman et al., 1988; Zhang, 1998). Modelling design Thus, while the IDEDØ model provides a basis it needs to be developed to development provide a more appropriate foundation for design performance performance measurement. The design activity uses resources to transform input to output under the direction of goals and constraints. Figure 2 illustrates such an activity and the 1205 key categories of knowledge that relate to it. All inputs and outputs may be represented as forms of knowledge, e.g. a designer is represented in this model as a knowledge resource (R), the state of the design prior to the activity may be described as the knowledge input (I), and so forth. Four categories of knowledge are identified here: (1) Knowledge input (I): the knowledge present prior to the activity. (2) Knowledge output (O): the knowledge present as a result of the activity taking place. (3) Knowledge goal (G): the knowledge which directs and constrains the activity. (4) Knowledge resource (R): the knowledge which acts on the input to produce the output. The category in which an element of knowledge resides is not fixed, but is derived from the context of the model, i.e. the activity to which it is related. For example, an output of one activity may act as a constraint on another. Although there is widespread use of efficiency and effectiveness to describe performance there are a variety of interpretations of these terms when applied in design and development. Efficiency () and effectiveness () are presented here as fundamental elements of performance which may be used to fully describe the phenomenon. That is: Design Development; Performance = Efficiency (); and Effectiveness (). A new model, E2, is presented here as a novel and unique means to formalise clearly the phenomenon of design performance and allow efficiency and effectiveness to be distinguished and related. Efficiency is related to input, output and resources, while effectiveness is determined by the relationship between output and goal(s). These elements are presented within the E2 model providing a fundamental representation of activity performance. 3.1 Efficiency In general, the efficiency of an activity is seen as the relationship (often expressed as a ratio) between what has been gained and the level of resource used. Assuming design as a knowledge processing activity (Ak) (Figure 3), the difference between the output (O) and the input (I) defines the knowledge gain from the activity (K+). The cost[3] of the activity may be determined by measuring the amount of resource knowledge used (RU). Therefore, the
IJOPM 22,11
1206 Figure 3. Efficiency ()
efficiency of this activity may be depicted as in Figure 3 and formulated as a ratio: ðAk Þ ¼ K þ : RU and K þ ¼ O I where: ðAk Þ I O Kþ RU
: efficiency ðÞ of an activity ðAk Þ; : input ðknowledgeÞ; : output ðknowledgeÞ; : knowledge gain; : resource ðknowledgeÞ used:
This formalism assumes that a quantitative comparison of the input and output knowledge can be carried out that results in a description of the level of knowledge gained in the activity. Similarly, it is assumed that the level of knowledge used in the activity may be measured and that the relationship between both quantities may be expressed in a meaningful form. However, measuring knowledge change is a non-trivial task (Kok, 2000). The nature and content of knowledge is often vague and poorly understood, leading to many of the problems associated with the allocation of the wrong metric. Knowledge engineering and management techniques (Hart, 1986) may provide a means to define and formalise the associated knowledge and possibly knowledge change. Having said this, often the nature of the knowledge itself is transformed through the activity (Kok, 2000). Consequently, given these difficulties, further research is required in this area to support activity performance measurement fully. In practice, a variety of metrics are used to determine efficiency, reflecting different aspects of the input, output or resource knowledge. For example, the cost of using a designer within an activity may be measured to reflect the amount of financial resource used in utilising this knowledge source. Efficiency of an activity is considered here to exist irrespective of whether it is measured or not, i.e. it is an inherent property of the activity. The selection and
application of metrics to determine efficiency allow particular views of Modelling design efficiency to be created, e.g. cost or time based efficiency. development 3.2 Effectiveness Activities are generally performed in order to achieve a goal, i.e. have a desired effect. However, the result obtained from performing an activity may not always meet the goal. The degree to which the result (output) meets the goal may be described as the activity effectiveness. Therefore, activity effectiveness, as depicted in Figure 4, can be expressed as:
performance 1207
ðAk Þ ¼ rC ðO; GÞ where: ðAk Þ rC O G
: effectiveness ðÞ of activity ðAk Þ; : relationship ðcomparativeÞ; : output ðknowledgeÞ; : goal ðknowledgeÞ:
This formalism assumes that the output knowledge (O) and goal knowledge (G) may be described in a manner which allows a direct comparison between them, and a relationship to be determined which indicates how closely they match. 3.3 Key performance indicators (KPIs) KPIs are often defined or determined in a ‘‘top-down’’ fashion and can relate to business or product development objectives (Griffin, 1996). Problems in practice of identifying appropriate KPIs often centre on being able to define the most relevant KPI to an activity or goal and to ensure alignment, congruency and inter-relations (coherence) of KPIs (Haffey and Duffy, 2001). A general problem of misdirected effort seems to arise from KPIs relating to the output of an activity rather than to its goal. The latter case is illustrated through an example, where the number of drawings (output) is used to measure the performance of a draughting activity. This then changes the behaviour and output of the activity in order to achieve a seemingly high performance with respect to the metric. However, the goal of the activity is more likely to be to define the product’s geometry to such a degree as to
Figure 4. Effectiveness ()
IJOPM 22,11
1208
enable further analyses or product development. The KPIs should in fact reflect and support the goals of the activities and not their output. A more appropriate performance indicator in this example may be something like drawing usability, appropriateness or completeness. Such measures could be given by downstream activities and fed back to indicate the drawing activity’s performance. Thus, it is argued that KPIs should reflect the goals of the activities and not the output. Such metrics will influence the behaviour of the activity, and hence the output, to achieve the desired performance, which will directly support the fulfilment of the goal. A number of KPIs can be defined for any specific activity to reflect the activity’s overall desired behaviour. 4. Design and management The knowledge goal (G) may be related to either the design (DG) e.g. reliability, aesthetics, or the design activity (DAG) involved in creating that design, for example time consumed, labour costs, resources consumed. The design and DAG may be managed intuitively by the designer in what has been presented in Figure 2 as one activity. However, it is proposed that there are two types of activity taking place: design activities (Ad) and design management activities (Am). Design activities are focused on the DGs while design management activities are concerned with DAGs and managing the trade-off between achieving design and DAGs to ensure best overall performance. At a design project level these activities are often defined separately and are generally carried out by different people, e.g. the designer or design team and the design manager. However, the distinction between these activity types exists even at the level of individual design activities. For example, during sketching a designer may glance at their watch to evaluate the time elapsed in relation to an implicit or explicit time goal before proceeding. This represents a change of activity, i.e. from a design activity focused on producing a sketch in accordance with a DG, to a design management activity focused on ensuring a DAG is achieved, e.g. sketch is completed on time. Given the basic design activity representation presented in Figure 2 and the distinction between design and design management presented above, a further model is introduced in Figure 5 to describe design and its management. This design activity management (DAM) model represents a managed activity, i.e. any activity in design aimed at achieving design and DAG. The categories of input (I), output (O), goal (G) and resource (R) knowledge, presented in Figure 2, are decomposed to reflect categories related to either design or design management activities as follows: I O G R
! ! ! !
DI and DAI DO and DAO DG and DAG DR and DAR
Modelling design development performance 1209
Figure 5. Design activity management (DAM) model
The managed activities described above are the fundamental elements of the design process, i.e. the design process consists of a number of managed activities with relationships such as those based on information dependencies and described as dependent, independent or interdependent (Eppinger et al., 1994). Having established the design and DAGs, perhaps through planning activities, the focus subsequently moves to ensuring these goals are achieved (Ullman et al., 1988), i.e. optimising overall effectiveness. This overall effectiveness is composed of design effectiveness, illustrating how well the DG have been met, and design management effectiveness, indicating if the DAGs, such as resource cost, have been met. In an informal sense, a designer will continually evaluate the effectiveness of his/her activities, e.g. checking their watch to assess time elapsed (design management effectiveness), or evaluating the aesthetic strengths of a particular concept (design effectiveness). More formally, effectiveness may be reviewed through simulating product behaviour and evaluating results at specific stages as represented within many of the phase models of the design process. 4.1 A model of performance measurement and management The measurement of design and design management effectiveness is presented here as a critical part of controlling a managed activity within a process model for performance measurement and management (PerMM) in design development.
IJOPM 22,11
1210
Figure 6. Design effectiveness
Figure 7. Effectiveness input
Figure 8. Design management effectiveness
The description below focuses on a typical sequence of events in evolving the state of the design highlighting the main decision points: . The design activity (Ad) takes DI as input and, directed by knowledge of the specific DG, produces an output (DO) aimed at meeting the goal. This output will be compared against the goal to determine the level of design effectiveness, (Ad), achieved in the activity (Figure 6). . The resulting level of design effectiveness (Ad) is used as an input of control knowledge into the design management activity (Figure 7). The description of design effectiveness may describe how well a design goal has been met or whether a constraint has been satisfied or not. . The design management activity analyses design management effectiveness, (Am), using knowledge (including meta-knowledge) of the resources being used in both the design and design management activities. This knowledge is primarily time and cost based, i.e. it refers to the time consumed or cost incurred during a particular activityresource relationship. This is compared against knowledge of the DAG, e.g. to achieve a design lead-time of one month, to determine the level of design management effectiveness (Figure 8).
.
Utilising design activity resource (DAR) knowledge the design Modelling design management activity evaluates the relationship between design and development design management effectiveness and decides on the controlling action, performance if any[4], which must be taken as an attempt to optimise overall effectiveness. This controlling action will typically involve changing the goals or resources in order to achieve a change in effectiveness.
Figure 9 is evolved from Figure 5 to illustrate the decision points and flow of control knowledge (shown as dashed lines) within a managed activity and serves to summarise the steps described above. That is, the model describes the process of measuring and managing performance in relation to both design and DAG. The following outlines the types of controlling action, aimed at optimising overall effectiveness, that may result from the evaluation of design and design management effectiveness: . At decision point ci the decision options are to terminate the activity having established satisfactory levels of design and design management effectiveness or to continue with the activity. .
.
1211
At decision point cj the decision options are to redefine goals and/or alter resource allocation. At decision point ck the decision options are to redefine DGs and/or DAGs. For example, the outcome of the design management activity
Figure 9. Performance measurement and management (PerMM) process model
IJOPM 22,11 .
1212
may be to set a new launch date for the project. In contrast, it may be more appropriate to reduce the targets specified in some DGs, e.g. life in service, while maintaining the original planned launch date. At decision point cl the decision options are to alter DRs and/or the DARs. For example, the outcome from the management activity may be to allocate additional design resources to achieve increased design effectiveness with a probable negative impact on design management effectiveness.
5. Relating efficiency and effectiveness Efficiency and effectiveness focus on related, yet contrasting performance elements. The efficiency is inherent in the behaviour of a particular activity and resource combination. It may be measured without any knowledge of the activity goals, although the goals may influence the behaviour of resources used in the activity and consequently the level of efficiency that results from their use. Effectiveness, in contrast, cannot be measured without specific knowledge of the activity goals. As is the case in measuring efficiency, the measurement of effectiveness involves the analysis of the activity output (O). However, effectiveness is obtained through analysing a specific element of the output knowledge, i.e. that which relates to the goal(s) of the activity. In certain cases there exists a direct relationship between effectiveness and efficiency. This relationship exists when the specific element of the output knowledge, which is evaluated to establish effectiveness, also describes an element of the resource used. For example, an activity may have a specific costrelated goal of minimising the activity cost, i.e. Gj: C = Min. Therefore, the element of the output knowledge (O) which must be evaluated is the cost knowledge (OC). However, determining the cost based efficiency of the activity also involves the analysis of cost incurred (RU-C) in carrying out the activity as part of the overall resources used (RU). In this particular instance the element of output knowledge used to establish effectiveness is the same as that used to establish efficiency. Therefore, an increase in the cost based efficiency of the activity will also result in an increase in the cost based effectiveness of the activity, given an activity goal of minimising cost. In cases such as this one the efficiency of the activity can provide insight into why a particular level of effectiveness has been obtained. In other cases a direct relationship between efficiency and effectiveness is not evident. Such cases exist where the specific element of the output knowledge that is evaluated to establish effectiveness has no relationship to the resource knowledge used in an activity. For example, where the goal of a design activity may be to maximise the dimensional accuracy of the artefact, G(s) = Max(s), the element of the output knowledge (O) which must be evaluated is the knowledge of the dimensional accuracy (O(s)). This knowledge provides no indication of the resource knowledge (R) used in the activity. Therefore, an increase in dimensional accuracy will give increased effectiveness with respect to this goal but there is no direct relationship with efficiency in this case.
6. Evaluation Modelling design The research presented in this paper has been evaluated as part of an overall development PhD project and is detailed O’Donnell (2000). A number of approaches have performance been taken in evaluating the work; only the first of these is described within the scope of this paper: . Worked example. Information from previously reported protocol studies 1213 (Delft Workshop) allowed a worked example to be developed, illustrating the application of the models. . Metrics review and analysis. A review of metrics presented throughout the literature on performance illustrated the applicability of the models presented here and assessed whether their principles were violated by existing metrics. . Expert appraisal. Presentation of the work to individuals from academia and industry with a view to obtaining a critical appraisal from experts in the area of design development, performance measurement and management, and risk management. . System development. The development of a computer-based system and approach (PERFORM), based on the work presented here, to support the analysis of influences on effectiveness in design development. . Industrial implementation. The implementation of the PERFORM approach within an industrial environment to assess the degree to which it is applicable within such an environment and its ability to identify areas for performance improvement. 6.1 Formalising Delft case study In developing a formalism of performance from the protocol studies two cases may be considered, i.e. the case of an individual designer (Case 1) versus the case of a team of three individuals (Case 2). The overall scope of the performance analysis is bounded by the initial description of the task, i.e. it involves the design and development of a concept in accordance with the brief given. The brief provides both background information on the state of the problem (i.e. current design state) which may be considered as input knowledge (I), while also providing detail on goals (G) such as the desired form of the product and the time allowed to carry out the task (two hours). The overall goal (GO) may, therefore, be seen as the delivery of a product concept in two hours. Both design and design management activities will take place over the twohour period. An overview of the DAM model as related to the Delft case is provided in Figure 10. The overall inputs, outputs, goals and resources are further detailed and related to design and/or design management activities as follows: . Inputs and outputs. The initial brief constitutes the state of the design at the start of the activity (DI) and the anticipated output at the end of the activity is a concept, represented by annotated sketches (DO). The brief
IJOPM 22,11
1214
Figure 10. DAM model of the Delft case
presents an isolated case and therefore knowledge of previous activity such as costs or time elapsed is not applicable as design activity input (DAI) at the beginning of the activity. However, as the activity progresses knowledge of the time elapsed becomes a key input to the design management activity, e.g. in one case a schedule is prepared which is subsequently monitored by the ‘‘timekeeper’’. .
.
Goals. The goals described within the brief refer only to the design (DG), i.e. ease of use, form and appearance, technical feasibility and price. In addition a time limit of two hours is set which may be categorised as a DAG. Resources. One of the key distinctions between the protocol data reported is in relation to the resources used to carry out the activities. Case 1 considers the completion of the task by an individual designer, while in Case 2 the task is completed by a team of designers. Where an individual designer is used, that individual constitutes both a design (DR) and a DAR as the individual will carry out the design activities to create the concept and the design management activities such as time management. In the data reported for the team case, a specific individual (timekeeper) was allocated to the task of time management, i.e. managing the design activity with respect to the time goal, in addition to contributing to the design. Therefore, two of the individuals constitute DR only, while the remaining designer constitutes both a design and DAR.
The above outlines key elements of a formalism which may be used to describe Modelling design performance of the activities and serve as a basis for analysing performance development and improving it. These elements are presented within Table II. It should be performance noted that the elements described here are those which are mentioned explicitly in the text. Additional elements are implicit within the case, for example, the timekeeper may use a watch as a DAR, the completeness of the information 1215 provided by the annotated sketches may be seen as a DG, or additional information elements may be represented in the output such as the approximate size (DO). 6.2 Measuring performance Performance may be measured through developing efficiency and effectiveness measures based on the understanding gained from formalising performance in the manner described above. For example, the efficiency could be determined through some comparison of the quantity of the information provided in the annotated sketches in relation to the time consumed and/or number of people used. That is, the knowledge gained (K+) in design development may be estimated through some assessment of completeness of the annotated sketches, e.g. a comparison of the number of information elements could be carried out. Information on the resources used is not readily available with the exception of Element
Case 1. Individual
Case 2. Team
Inputs DI1 DAI1
Design brief No initial input
Design brief No initial input
Concept description including information on: Ease of use Appearance Technical feasibility Estimated price Time elapsed
Concept description including information on: Ease of use Appearance Technical feasibility Estimated price Time elapsed
Ease of use Sporty appearance Appealing form Technically feasible Reasonable price Complete within two hours
Ease of use Sporty appearance Appealing form Technically feasible Reasonable price Complete within two hours
Individual designer
Team member 1 Team member 2 Team member 3 (part) Information resources via EXP1 Team member 3 (‘‘timekeeper’’)
Outputs DO1 DO2 DO3 DO4 DAO1 Goals DG1 DG2 DG3 DG4 DG5 DAG1 Resources DR1 DR2 DR3 DR4 DAR1
Information resources via EXP1 Individual designer
Table II. Elements of performance in the Delft study
IJOPM 22,11
the time and number of people in each case and therefore efficiencies may be measured and compared as follows: Efficiency in terms of time: time ¼ No: of information elements : time used
1216
Efficiency in terms of people: people ¼ No: of information elements : no: of people From the information made available on the experiments it is not possible to distinguish the efficiency of the design activity from that of the design management activity. However, if the time used in checking and managing the schedule were recorded then the efficiency of this management activity could be analysed. An example of a measure would be to compare the number of scheduling related instructions (DAO) and the time required to define them (DAR). The design effectiveness may be determined from a comparison of the information provided in the sketches with the specific DGs. The design management effectiveness may be determined through relating the time elapsed to the time goal (DAG). The effectiveness of design development in each case may be determined in relation to the goals as follows: Effectiveness in terms of ease of use: ease of
use
¼ rC ðDO1 ; DG1 Þ
Effectiveness in terms of appearance: appearance ¼ rC ðDO2 ; DG2 Þ Effectiveness in terms of form: form ¼ rC ðDO3 ; DG3 Þ Effectiveness in terms of feasibility: feasibility ¼ rC ðDO4 ; DG4 Þ Effectiveness in terms of price: price ¼ rC ðDO5 ; DG5 Þ Effectiveness in meeting time goal: time ¼ rC ðDAO1 ; DAG1 Þ The manner in which the DGs are specified makes such comparisons difficult, e.g. the appearance of the product is specified as ‘‘sporty’’ and ‘‘appealing’’. Assessing the degree to which these goals are met is likely to require subjective measures. Similarly, the specification of the price as within a ‘‘reasonable
range’’ requires further clarification for a more accurate analysis of Modelling design performance. This highlights both the difficulty in analysing performance at development early stages in design where specific goals are difficult to determine and the performance need to invest time in improving clarity where possible. Applying efficiency and effectiveness metrics as described above provides an assessment of performance of the activities once the experiment is completed.
1217
6.3 Summary Although the example given above is somewhat simplistic, it illustrates how the work presented in the paper may be applied. The modelling formalisms presented here allow a model to be developed that describes the key elements of performance in the Delft case and allows efficiency and effectiveness to be distinguished and measures to be established. The evaluation, therefore, supports the ability of the formalisms to represent an actual design scenario. It is possible to identify and distinguish design and design management activities within the experiments, i.e. the actual design of the artefact and time keeping, supporting the principles described in the DAM and PerMM models. However, from the information available on the Delft experiments it is not possible to readily identify measures which distinguish the efficiency of the design activity from that of the design management activity. This may not be significant here due to the somewhat artificial environment resulting in comparatively little time spent on design management as the management task is a relatively small one (i.e. mainly timekeeping). However, this could be a critical area in larger projects where design management requires considerable resource. The efficiency of the overall managed activity could be considered to be the relationship between the completeness of the information provided in the concept and the time taken (assumed to be two hours). Similarly, the overall effectiveness could be assessed as the degree to which the overall goal of producing a design concept within two hours was met, i.e. it will be a combination of the design and design management effectiveness. 7. Discussion 7.1 Axioms of performance From the work presented here and elsewhere (O’Donnell, 2000) the following axioms[5] of performance can be argued: . Axiom 1. Activity performance. Activities are the fundamental means that create performance. Activities are the fundamental elements that transform input to output and are the basic components of processes, phases, projects, and higher level activities. All other aspects of performance support and influence its behaviour, but its actual creation is through an activity. That is, no other aspect related to performance creates it. Other aspects influence the type, definition and behaviour of an activity but it is the activity itself that realises performance.
IJOPM 22,11
.
1218 .
Axiom 2. Efficiency and effectiveness. All performance can be measured by efficiency and/or effectiveness. That is, no matter the metric(s) or aspect(s) under consideration, all indicators of performance, no matter how general or specific, will indicate either an efficiency or effectiveness measure. Thus, while there may be multiple metrics for a particular activity or process (amalgamation of activities) they can be categorised into two fundamental types of efficiency and/or effectiveness. Axiom 3. Activity and management. Activities and their management are inextricably linked. Carrying out an activity will always involve and element of management. Thus, every activity, even at an individual cognitive level, will involve its management (see section 4). Performance measurement must ensure that the correct metric is being used for the actual activity or its management. Conflicts and misdirected effort ensues if these are not clearly and correctly identified.
It is suggested that some of the fundamental axioms of performance are either neglected or misunderstood when considering performance management practice. 7.2 The E2 model The E2 model provides a consistent foundation and fundamental basis to ensure their consideration and of the key aspects of an activity’s performance, and in particular: . Identify the most appropriate KPIs for a particular activity or process. . Ensure alignment, congruency and coherence between activities and their performance. . Provide the key elements (goals, inputs, resources and outputs: GIROs) for defining KPIs relevant to an activity’s desired performance (goal centred) rather than output centred. . Puts activities as the central driving force of performance. . Supports a consistent understanding of the relations between performance, activities and their main elements (GIRO). In summary, the E2 model overcomes the main shortcomings as identified in section 2.2, by providing a consistent definition of the key elements of performance, distinguishing the differences and relation between efficiency and effectiveness, and provides a means to measure performance itself and not its influencing factors. 8. Conclusion The treatment of design performance within existing research lacks clarity on the nature of performance itself as a basic foundation for work in this area. The E2 model clearly distinguishes and relates the key elements of performance, efficiency () and effectiveness (). Further, influences on performance may be
clearly distinguished, e.g. flexibility may be viewed as the ability of a designer Modelling design to adapt to changing goals. This influences the performance of the activity but development is not in itself a measure of performance. Three axioms of performance state performance that activities are the fundamental means that create performance, activities and their management are inextricably linked, and that all metrics can be typified to efficiency or effectiveness indicators. 1219 The DAM and PerMM models distinguish and relate the performance of the design (artefact) and the design activity and illustrate the nature of performance measurement and management in design development. Evaluation of the work through a number of methods (O’Donnell, 2000) has highlighted its ability to provide a new insight and understanding of performance in general, with particular relevance to the design development process. However, from the case study presented it was not possible to distinguish the efficiency of the design activity from that of the design management. In addition, efficiency would seem to be an inherent feature of an activity and one that is difficult to measure given the problems with determining the difference between the input and output levels of knowledge. This serves as a basis for continuing research in design development performance. Notes 1. Here product development is taken to include all life-phases of developing a product including customer requirements specification, conceptualisation (design), manufacturing (production, assembly, etc.) and operational corrections/modifications. 2. The table is not intended to represent an exhaustive coverage of all terms and definitions, but presents a sample to indicate the range of interpretations. 3. Cost is used here as a general metric to describe the level of time, money, material, etc. used in the activity. 4. It may be desirable to take no controlling action, i.e. to maintain all goals and resources as they currently are and allow the managed activity to continue. 5. Self-evident truths from Fowler and Fowler (1995). References Andreasen, M.M. (1994), ‘‘Modelling – the language of the designer’’, Journal of Engineering Design, Vol. 5 No. 2, pp. 103-15. Andreasen, M.M. and Hein, L. (1987), Integrated Product Development, IFS Publications and Springer-Verlag, London. Brookes, N.J. and Backhouse, C.J. (1998), ‘‘Measuring the performance of product introduction’’, Journal of Engineering Manufacture, Vol. 212 No. 1, pp. 1-11. Brown, S.L. and Eisenhardt, K.L. (1995), ‘‘Product development: past research, present findings, and future directions’’, Academy of Management Review, Vol. 20 No. 2, pp. 343-78. Chang, Z.Y. and Yong, K.C. (1991), ‘‘Dimensions and indices for performance evaluation of a product development project’’, International Journal of Technology Management, Vol. 6 No. 1/2, p. 155-67. Christainsen, R.T. (1993), Modelling Efficiency and Effectiveness of Coordination in Engineering Design Teams, Stanford University, Stanford, CA.
IJOPM 22,11
1220
Clark, K.B. and Fujimoto, T. (1991), Product Development Performance: Strategy, Organisation and Management in the World Auto Industry, Harvard Business School Press, Boston, MA. Colquhoun, G.J., Baines, R.W. and Crossley, R. (1993), ‘‘A state of the art review of IDEF0’’, International Journal of Computer Integrated Manufacturing, Vol. 6 No. 4, pp. 252-64. Cordero, R. (1989), ‘‘The measurement of innovation performance in the firm: an overview’’, Research Policy, Vol. 19, pp. 185-92. Cross, N., Christiaans, H. and Dorst, K. (eds) (1996), Analysing Design Activity, John Wiley & Sons, Chichester, p. 463. Doz, Y. (1996), ‘‘New product development effectiveness: a triadic comparison in the information technology industry’’, in Nishiguchi, T. (Ed.), Managing Product Development, Oxford University Press, Oxford, pp. 13-33. Duffy, A.H.B. (1998), ‘‘Design productivity’’, in Duffy, A.H.B. (Ed.), The Design Productivity Debate, Springer-Verlag Publications. Dwight, R. (1999), ‘‘Searching for real maintenance performance measures’’, Journal of Quality in Maintenance Engineering, Vol. 5 No. 3, pp. 258-75. Emmanuelides, P.A. (1993), ‘‘Towards an integrative framework of performance in product development projects’’, Journal of Engineering and Technology Management, Vol. 10, pp. 363-92. Eppinger, S.D., Whitney, D.E., Smith, R.P. and Gebala, D.A. (1994), ‘‘A model-based method for organising tasks in product development’’, Research in Engineering Design, Vol. 6 No. 1, pp. 1-13. Fowler, H.W. and Fowler, F.G. (Eds) (1995), The Concise Oxford Dictionary, 9th ed., Clarendon Press, Oxford. Goldschmidt, G. (1995), ‘‘The designer as a team of one’’, Design Studies, Vol. 16 No. 2. Griffin, A. (1996), The Impact of Engineering Design Tools on New Product Development Efficiency and Effectiveness, Institute for the Study of Business Markets, The Pennsylvania State University, University Park, PA. Griffin, A. and Page, A.L. (1993), ‘‘An interim report on measuring product development success and failure’’, Journal of Product Innovation Management, Vol. 10, pp. 291-308. Griffin, A. and Page, A.L. (1996), ‘‘PDMA success measurement project – recommended measures for product development success and failure’’, Journal of Product Innovation and Management, Vol. 13, pp. 478-96. Haffey, M.K.D. and Duffy, A.H.B. (2001), ‘‘Modelling performance behaviour’’, 1st International Workshop on Performance Measurement, IFIP 5.7 Special Interest Group on Performance Measurement, 24-25 May, University of Strathclyde, Glasgow. Hales, C. (1993), Managing Engineering Design, Longman Scientific and Technical, Harlow. Hart, A. (1986), Knowledge Acquisition for Expert Systems, Kogan Page Publishers, New York, NY. Johnson, H.T. and Kaplan, R.S. (1987), Relevance Lost – the Rise and Fall of Management Accounting, Harvard Business School Press, Boston, MA. Kok, S.S. (2000), ‘‘Modelling learning in design’’, Department of Design, Manufacture and Engineering Management, University of Strathclyde, Glasgow. McDonough, E.R. and Griffin, A. (1996), The Impact of Organisational Tools on New Product Development Efficiency and Effectiveness, Institute for the Study of Business Markets, The Pennsylvania State University, University Park, PA. McGrath, M.E. (1994), ‘‘The R&D effectiveness index: metric for product development performance’’, Journal of Product Innovation Management, Vol. 11, pp. 201-12.
Meyer, M.W. and Gupta, V. (1994), ‘‘The performance paradox’’, Research in Organisational Behaviour, Vol. 16, pp. 309-69. Montoya-Weiss, M.M. and Calantone, R. (1994), ‘‘Determinants of new product performance: a review and meta-analysis’’, Journal of Product Innovation Management, Vol. 11, pp. 397-417. Moseng, B. and Bredrup, H. (1993), ‘‘A methodology for industrial studies of productivity performance’’, Production Planning and Control, Vol. 4 No. 3. Neely, A. (1999), ‘‘The performance measurement revolution: why now and what next?’’, International Journal of Operations & Production Management, Vol. 19 No. 2, pp. 205-28. Neely, A., Gregory, M. and Platts, K. (1995), ‘‘Performance measurement system design: a literature review and research agenda’’, International Journal of Production & Operations Management, Vol. 15 No. 4, pp. 80-116. Neely, A., Mills, J., Gregory, M., Richards, H., Platts, K. and Bourne, M. (1996), Getting the Measure of Your Business, Findlay Publications, Horton Kirby. O’Donnell, F.J. (2000), ‘‘A methodology for performance modelling and analysis in design development, in design manufacture and engineering management’’, PhD thesis, CAD Centre, Department of Design Manufacture and Engineering Management, University of Strathclyde, Glasgow, December. Pahl, G. and Beitz, W. (1996), Engineering Design: A Systematic Approach, 2nd rev. ed., Wallace. K. (Ed.), Springer-Verlag, London. Persidis, A. and Duffy, A. (1991), ‘‘Learning in engineering design’’, IFIP TC/WG5.2 Third Intelligent Workshop on Computer Aided Design, Elsevier Science Publishers B.V., NorthHolland, Amsterdam. Pritchard, R.D. (1990), Measuring and Improving Organisational Productivity: A Practical Guide, Praeger Publishers, New York, NY. Radcliffe, D.F. (1996), ‘‘Concurrency of actions, ideas and knowledge within a design team’’, in Cross, N., Christiaans, H. and Dorst, K. (Eds), Analysing Design Activity, John Wiley & Sons, New York, NY. Rolstadas, A. (1998), ‘‘Enterprise performance measurement’’, International Journal of Operations & Production Management, Vol. 18 No. 9/10, pp. 989-99. Sinclair, D. and Zairi, M. (1995), ‘‘Effective process management through performance measurement: Part 3 – in integrated model of total quality-based performance measurement’’, Business Process Re-engineering and Management Journal, Vol. 1 No. 3, pp. 50-65. Ullman, D.G., Dietterich, T.G. and Stauffer, L.A. (1988), ‘‘A model of the mechanical design process based on empirical data’’, AI EDAM, Vol. 2 No. 1, pp. 33-52. van Drongelen, C.K. and Cook, A. (1997), ‘‘Design principles for the development of measurement systems for research and development processes’’, R&D Management, Vol. 27 No. 4, pp. 345-57. Zhang, Y. (1998), ‘‘Computer-based modelling and management for current working knowledge evolution support’’, Department of Design, Manufacture and Engineering Management, University of Strathclyde, Glasgow.
Modelling design development performance 1221
The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/0144-3577.htm
IJOPM 22,11
1222
A framework of the factors affecting the evolution of performance measurement systems Mike Kennerley and Andy Neely Centre for Business Performance, Cranfield School of Management, Cranfield, UK Keywords Performance measurement, Development, Organizational change Abstract The effectiveness of performance measurement is an issue of growing importance to industrialists and academics alike. Many organisations are investing considerable amounts of resource implementing measures that reflect all dimensions of their performance. Consideration is being given to what should be measured today, but little attention is being paid to the question of what should be measured tomorrow. Measurement systems should be dynamic. They have to be modified as circumstances change. Yet few organisations appear to have systematic processes in place for managing the evolution of their measurement systems and few researchers appear to have explored the question, what shapes the evolution of an organisation’s measurement system? The research reported in this paper seeks to address this gap in the literature by presenting data that describes the forces that shape the evolution of the measurement systems used by different organisations.
Introduction Although it has long been recognised that performance measurement has an important role to play in the efficient and effective management of organisations, it remains a critical and much debated issue. Significant management time is being devoted to the questions – what and how should we measure – while substantial research effort, by academics from a wide variety of management disciplines, is being expended as we seek to enhance our understanding of the topic and related issues (Neely, 1999). Survey data suggest that between 40 and 60 per cent of companies significantly changed their measurement systems between 1995 and 2000 (Frigo and Krumwiede, 1999). Most of these initiatives, however, appear to be static. Although many organisations have undertaken projects to design and implement better performance measures, little consideration appears to have been given to the way in which measures evolve following their implementation (Waggoner et al., 1999). It is important that performance measurement systems be dynamic, so that performance measures remain relevant and continue to reflect the issues of importance to the business (Lynch and Cross, 1991). International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, pp. 1222-1245. # MCB UP Limited, 0144-3577 DOI 10.1108/01443570210450293
The authors are grateful to the Engineering and Physical Sciences Research Council (EPSRC) for the award of research grant number GR/K88637, to carry out the research reported in this paper.
In order to ensure that this relevance is maintained, organisations need a process in place to ensure that measures and measurement systems are reviewed and modified as the organisation’s circumstances change (Dixon et al., 1990). Yet few organisations appear to have systematic processes in place for managing the evolution of their measurement systems and few researchers appear to have explored the question – what shapes the evolution of an organisation’s measurement system. The research reported in this paper seeks to address this gap in the literature by presenting a framework that describes the forces that shape the evolution of the measurement systems used by different organisations. Following this introduction the paper consists of a further six sections. The next section discusses the literature regarding the evolution of performance measurement systems, providing the context for the research. Descriptions of the research methodology, the case study findings and the resultant framework of factors affecting the evolution of performance measures are then presented. The subsequent discussion is followed by conclusions that are drawn in the final section. Performance measurement literature The problem of how organisations should assess their performance has been challenging management commentators and practitioners for many years. Financial measures have long been used to evaluate performance of commercial organisations. By the early 1980s, however, there was a growing realisation that, given the increased complexity of organisations and the markets in which they compete, it was no longer appropriate to use financial measures as the sole criteria for assessing success. Following their review of the evolution of management accounting systems, Johnson and Kaplan highlighted many of the deficiencies in the way in which management accounting information is used to manage businesses (Johnson, 1983; Kaplan, 1984; Johnson and Kaplan, 1987). They highlighted the failure of financial performance measures to reflect changes in the competitive circumstances and strategies of modern organisations. While profit remains the overriding goal, it is considered an insufficient performance measure, as measures should reflect what organisations have to manage in order to profit (Bruns, 1998). Cost focused measurement systems provide a historical view, giving little indication of future performance and encouraging short termism (Bruns, 1998). The shortcomings of traditional measurement systems have triggered a performance measurement revolution (Eccles, 1991; Neely, 1999). Attention in practitioner, consultancy and academic communities has turned to how organisations can replace their existing, traditionally cost based, measurement systems with ones that reflect their current objectives and environment. Many authors have focused attention on how organisations can design more appropriate measurement systems. Based on literature, consultancy experience and action research, numerous processes have been developed that organisations
Performance measurement systems 1223
IJOPM 22,11
1224
can follow in order to design and implement performance measurement systems (Bourne et al., 1999). Many frameworks, such as the balanced scorecard (Kaplan and Norton, 1992), the performance prism (Kennerley and Neely, 2000), the performance measurement matrix (Keegan et al., 1989), the results and determinants framework (Fitzgerald et al., 1991), and the SMART pyramid (Lynch and Cross, 1991) have been proposed that support these processes. The objective of such frameworks is to help organisations define a set of measures that reflects their objectives and assesses their performance appropriately. The frameworks are multidimensional, explicitly balancing financial and nonfinancial measures. Furthermore, a wide range of criteria has also been developed, indicating the attributes of effective performance measures and measurement systems. These include the need for measures to relate directly to the organisation’s mission and objectives, to reflect the company’s external competitive environment, customer requirements and internal objectives (Globerson, 1985; Wisner and Fawcett, 1991; Maskell, 1989; Kaplan and Norton, 1993). Others make explicit the need for strategies, action and measures to be consistent (Lynch and Cross, 1991; Dixon et al., 1990). At the heart of the processes, frameworks and criteria discussed, as with much that has been written on the subject of performance measurement, is the premise that measures and measurement systems must reflect the context to which they are applied (Neely, 1999). Indeed as Johnson (1983) observed, the introduction of financial performance measures, such as cash flow and return on investment, reflected the changing marketplace in which organisations competed. At the turn of the century sole traders were giving way to owner managers who needed to assess the return on their investment in plant and premises. The performance measurement revolution has prompted many organisations to implement new performance measurement systems, often at considerable expense. However, unlike the environment in which organisations operate, many measurement initiatives appear to be static. Senge (1992) argues that, in today’s complex business world, organisations must be able to learn how to cope with continuous change in order to be successful. Eccles (1991) suggests that it will become increasingly necessary for all major businesses to evaluate and modify their performance measures in order to adapt to the rapidly changing and highly competitive business environment. Numerous authors espouse the need for reflection on measures to ensure that they are updated to reflect this continuous change (Meyer and Gupta, 1994; Ghalayini and Noble, 1996; Dixon et al., 1990; Wisner and Fawcett, 1991). However, there has been little evidence of the extent or effectiveness with which this takes place. Moreover, the literature suggests that ineffective management of the evolution of measurement systems is causing a new measurement ‘‘crisis’’, with organisations implementing new measures to reflect new priorities but failing to discard measures reflecting old priorities resulting in uncorrelated and inconsistent measures (Meyer and Gupta, 1994). Furthermore, it is suggested
that organisations are drowning in the additional data that is now being collected and reported (Neely et al., 2000). As with measurement systems introduced at the turn of the century, there is a danger that failure to manage effectively the way in which measurement systems change over time will cause new measurement systems to lose their relevance, prompting a new crisis and necessitating a further measurement revolution. This raises a crucial question. Why do performance measurement systems fail to change as organisations change, rendering them irrelevant? This is an important question to answer if history is not to be repeated and organisations are to avoid the expense of another extensive overhaul of their measurement systems. Wisner and Fawcett (1991) acknowledge the need for performance measures to be reviewed and changed to ensure that measures remain relevant in the last step of their nine step process. They highlight the need to ‘‘re-evaluate the appropriateness of the established performance measurement systems in view of the current competitive environment’’. Bititci et al. (2000) identify the need for performance measurement systems to be dynamic to reflect changes in the internal and external environment; review and prioritise objectives as the environment changes; deploy changes in objectives and priorities; and ensure gains achieved through improvement programmes are maintained. Dixon et al. (1990) and Bititci et al. (2000) propose audit tools that enable organisations to identify whether their existing measurement systems are appropriate given their environment and objectives. Bititci et al. (2000) go on to posit that a dynamic performance measurement system should have: . an external monitoring system, which continuously monitors developments and changes in the external environment; . an internal monitoring system, which continuously monitors developments and changes in the internal environment and raises warning and action signals when certain performance limits and thresholds are reached; . a review system, which uses the information provided by internal and external monitors and the objectives and priorities set by higher level systems, to decide internal objectives and priorities; and . an internal deployment system to deploy the revised objectives and priorities to critical parts of the system. Bourne et al. (2000) suggest measurement systems should be reviewed and revised at a number of different levels. They identify the need for review of targets and performance against them; individual measures as circumstances change; and the set of measures to ensure that they reflect the strategic direction. Although the authors discussed above propose the need to review measures and suggest techniques for such review, there is little discussion of their
Performance measurement systems 1225
IJOPM 22,11
1226
application in practice, investigation of how measures actually change or of the factors that affect how effectively and efficiently performance measurement systems change. With a few notable exceptions (Meyer and Gupta, 1994; Townley and Cooper, 1998; Bourne et al., 2000), empirical investigation of the evolution of measurement systems over time remains a considerable gap in performance measurement research (Neely, 1999). Meyer and Gupta (1994) observe that measures tend to lose their relevance and ability to discriminate between good and bad performance over time as performance objectives are achieved or as behaviour no longer reflects the performance objectives underpinning the measures. They observe that failure to effectively manage this change causes the introduction of new measures ‘‘that are weakly correlated to those currently in place’’ so that an organisation will have a diverse set of measures that do not measure the same thing. Townley and Cooper (1998) undertook a longitudinal study of performance measurement in Alberta government in Canada. They observed that support for performance measurement can diminish over time. They observe that measurement initiatives can suffer from loss of initial enthusiasm, which is replaced by scepticism and disillusionment. They cited a number of causes of this including failure to manage the change appropriately, underestimating the effort required and lack of commitment to the change. They also identified that political issues and the involvement of employees affect success. Not only does their study identify factors affecting the success of performance measurement activities, but it also highlights the need for support of such activities within the organisation. In a case study company, Bourne et al. (2000) observed that performance measures changed over time. They identified that changes were prompted by existing budgetary review processes, chance, intervention of the researchers and eventually by design, however they provide little insight into how this change by design took place. Despite the limited discussion of evolution in the performance measurement literature, it is possible to draw lessons from a variety of other streams of literature that address issues relating to the management of change (Waggoner et al., 1999). Based on a review of the relevant literature, Waggoner et al. (1999) summarise the key forces driving and demanding change as: customers, information technology, the marketplace, legislation (public policy), new industries, nature of the work (e.g. outsourcing) and future uncertainty. However, many authors also identify barriers to change that have received little attention in the performance measurement literature. Gabris (1986) identifies four categories of such barriers: (1) process burden, where processes such as performance measurement take employees away from their actual responsibilities; (2) internal capacity, where organisations lack the in-house capability to support an initiative;
(3) credibility anxiety, where organisations suffer from an overload of management techniques; and (4) the ‘‘Georgia giant syndrome’’, where management techniques work only under rigorous and closely supervised control conditions. These factors can be considered to be the organisation’s readiness for change (Waggoner et al., 1999). Furthermore, Kotter (1996) argues that willingness or urgency to change throughout the organisation is necessary for such change to be effective. Greiner (1996) categorises inhibiting factors as institutional, pragmatic, technical and financial. Numerous authors (such as Scott, 1995 and Pettigrew and Whipp, 1991) also highlight that the political nature of organisations requires further consideration, one of a number of factors demonstrating the impact that corporate culture can have on evolutionary change (Tichy, 1983). The literature reviewed highlights the importance of managing the evolution of performance measurement systems to ensure that they continue to reflect the environment and objectives of the organisation. The literature suggests that the factors affecting evolutionary change within organisations, and hence the evolution of performance measures, are many and complex. However, these issues can be grouped into two main themes:
Performance measurement systems 1227
(1) drivers of change (those factors that cause change to be necessary); and (2) barriers to change (those factors that must be overcome if change is to be effective). These issues are summarised in Figure 1. The research reported in this paper seeks to investigate these drivers of, and barriers to, evolution as they apply to performance measurement systems. Methodology A multiple case study approach was used to investigate the way in which performance measures actually evolve within organisations. The research involved semi-structured interviews with a total of 25 managers from a range
Figure 1. Summary of factors affecting evolution drawn from the literature
IJOPM 22,11
1228
of management functions, from seven different organisations. The companies involved in the research were from the industries shown in Table I. The interview structure was designed to investigate the key themes identified from the literature reviewed. As such the case studies sought to answer the following questions: . What factors encourage the introduction of new measures, modification of existing measures and deletion of obsolete measures? . What factors inhibit the introduction of new measures, modification of existing measures and deletion of obsolete measures? The companies were selected on the basis of their considerable experience in the implementation and use of performance measures. Companies from different industry sectors and with a wide variety of competitive and organisational characteristics were deliberately chosen to introduce diversity into the sample. This enabled the identification of factors affecting evolution of measurement in a variety of different circumstances. Similarly, interviewing managers from a number of different departments ensured that consideration was given to the diversity of factors affecting evolution in different functional circumstances. As a result the findings of the case studies provide a broad understanding of the factors affecting the evolution of an organisation’s performance measures. Case study findings There was general consensus among the managers interviewed of the need for performance measures to evolve over time, so that they reflect the changing circumstances and requirements of the organisation. The group technical and quality director of company 7 stated that: ‘‘ensuring that performance measures continue to reflect the issues that are important to the organisation is important if measurement is to be useful and help management’’. The consultancy sales manager of company 2 stated that: ‘‘the evolution of measures ensures that they increase in sophistication and change to reflect the changes in behaviour we want to achieve’’. The systems analyst from company 6 indicated that: ‘‘evolution enables us to tackle soft issues and develop hard measures to reflect how well we are doing’’.
Table I. Companies involved in the research and their industry
Company
Industry
1 2 3 4 5 6 7
Maintenance of transport infrastructure Supplier of IT services Supplier of stationary to retail and commercial sectors Courier/global package delivery Utility – energy generation and supply Manufacturer of food packaging Manufacturer and supplier of printing machinery
Although the need for performance measures to evolve over time was recognised, the evolution of measures was managed with varying degrees of success. Findings from each of the organisations are now discussed in turn. Company 1 In the past, company 1 had been unable to manage effectively the evolution of performance measures. The lack of flexible information systems and inertia throughout the organisation were found to be the main barriers to the effective management of legacy measurement systems. The problems prompted a company-wide initiative to establish effective performance measurement, explicitly addressing problems that had previously been experienced. The managing director was the major driving force behind the initiative and extensive use was made of existing and accepted communication tools to ensure performance measurement had the appropriate credibility. As the human resources manager remarked: ‘‘Effective use of the measurement system is due to the managing director’s promotion of the need for and importance of measurement and his use of measurement to manage and communicate’’. The managing director highlighted the need for flexible systems ‘‘None of the commercial performance measurement software provided the required support – you must have a system that satisfies your requirements’’. In-house information systems were developed to provide data collection, analysis and reporting systems giving flexibility not provided by systems available on the market. Addressing these issues, and integrating performance measurement into the strategy development and review process, provided the organisation with a measurement system that they believed would evolve with the business’s requirements. Company 2 Although performance measurement systems had been implemented in company 2 for a number of years, failure to actually use new performance measures to manage the business was seen as major barrier to their deployment and hence evolution. Although senior management had backed the implementation of a balanced set of measures, the continued emphasis on financial performance measures prevented use of the balanced measurement system being embedded throughout the organisation. As in company 1, company 2 used experiences of ineffective measurement practices in the past to design a measurement system with the attributes that they considered necessary to maintain a relevant set of performance measures in the future. To ensure that their measures remained relevant, managers in company 2 explicitly included a review of measures in the periodic review of business processes. The head of business process development highlighted the importance of having the appropriate systems to facilitate measurement activity and the evolution of measurement systems. ‘‘New systems have been designed from scratch to be flexible enabling measures to be changed easily. The system being Web-based enables worldwide access to all information allowing
Performance measurement systems 1229
IJOPM 22,11
1230
information sharing. This facilitates benchmarking and the transfer of best practice. The global availability of the same reporting systems enables commonality of approach’’. Furthermore he highlighted that: ‘‘reporting needs to be efficient to reduce the resources required to administer measurement, allowing resources to be dedicated to acting on the results.’’ The system was designed to enable efficient and effective data collection and reporting, minimising the effort of measurement to ensure acceptance throughout the organisation. According to the consultancy sales manager: ‘‘Benchmarking of performance against competitors (including those in new markets) has given a common understanding of the need to improve and where improvement should be focused. This has reduced any resistance to the change of performance measures as the need can be demonstrated.’’ This enabled the organisation to overcome some of the people issues that had limited the development of performance measurement activities in the past. Company 3 The evolution of measures was not effectively managed in company 3. ‘‘The culture at [company 3] is a barrier to the implementation of a consistent approach to measurement across the whole company.’’ The ad hoc approach to performance measurement that was adopted led to inconsistency in approaches between different business units and geographical locations. The inconsistency in measurement practices limited the comparability of performance data, detrimentally affecting the credibility, and hence acceptance, of performance measures. Despite attempts to change measures to reflect changing business circumstances, managers were reluctant to use non-financial data to manage the business. ‘‘The overriding factor affecting the acceptance of performance measurement is that it become a business issue so that it occupies the minds of managers and measures are used to manage the business’’ (Manager – Stationary Office Supplier). This reflects the need for managers to actively use measures to manage the business. It was found that this would increase their desire to ensure measures changed to remain appropriate, as their performance would be assessed on them. Inflexible IT systems were also found to be a major barrier to evolution. The European customer care manager specifically noted that: ‘‘it is not possible to change the structure and content of the performance reports produced by the mainframe IT system.’’ Company 4 The use of performance measurement to manage the business was accepted in company 4. However, the tendency to report too much data and produce too many measurement reports acted as a significant barrier to evolution. The service recovery manager stated: ‘‘I spend too much time preparing reports for my manager to take to board meetings. It prevents me from reviewing and updating measures so that they remain current. Most of the reports are never
referred to, they are just a security blanket in case he is ever asked to produce the data.’’ In the past key individuals had stood in the way of the use of some measures. ‘‘This resistance was due to reluctance to provide a better understanding of actual performance for which they were responsible. Removal of the individuals has been the most successful way of eliminating the problem’’ (Service Recovery Manager). The availability of people with the appropriate skills to analyse and redefine measures was also identified as an issue. This was particularly the case when individuals responsible for measurement left departments or the company all together. It was recognised that measurement practices could be developed further by planning skills development and ensuring that the appropriate skills were maintained in the areas they were required. Company 4 also provided an example of the effect of the design of measures on their use. While discussing the graphical representation of one measure, the field service manager explained: ‘‘nobody uses this measure as they don’t understand it. I would explain it to you but I don’t understand it either’’. As a result the measure was not seen as relevant and was not used. Company 5 Extensive performance measurement implementation had been undertaken in company 5. However, as in company 2, although senior management had initiated the implementation of new measures, they failed to use the resultant performance measurement data, in favour of traditional financial performance measures. ‘‘The previous CEO paid lip service to the scorecard but only really focussed on the financials, hence this is where all attention was focused’’ (Head of Strategic Planning). As a result the new measures were not considered to be important at other levels of the organisation and they were not effectively used. Measurement reverted to financial measurement and the process of evolution was stifled. This clearly demonstrated the need for top level support for measurement and the need for a change in mindset of management so that measures are used to manage the business. Company 6 Company 6 provided the best example of managing the evolution of measurement systems. The primary factor facilitating evolution was the availability of resources dedicated to measurement and the management of performance measures. ‘‘The availability of a dedicated employee who is responsible for the review of measures enables gaps to be identified and the need to change existing measures as well as identifying performance measures’’ (Sales Order Manager). The dedicated systems analyst ensured that measures were reviewed and that action was taken to improve performance and ensure that measures were changed to remain relevant. In addition, ‘‘having split responsibility and budget from operations and the IT department enables me to develop systems that would not be justified under either department individually’’. This ensured
Performance measurement systems 1231
IJOPM 22,11
1232
that systems were flexible enough to change as required. The availability of a manager dedicated to measurement, who had credibility within all areas of the business stimulated measurement activity and helped overcome barriers to the acceptance and evolution of measurement, such as inflexible payroll structures and high staff turnover. Company 6 highlighted the need to create the appropriate environment in which the use of performance measures is most effective. Weekly meetings to review performance were open and honest discussions of performance, including new issues requiring measurement and identifying new areas of performance on which to focus improvement attention. ‘‘It is important to recruit and retain employees who are open to new ideas and are willing and able to implement new performance measures.’’ ‘‘Use of neutral measures, that focus on improvement and do not apportion blame, help acceptance and adoption of measures.’’ Company 7 The lack of a formal review process was considered to be the main reason that the evolution of performance measures was not managed in company 7 (‘‘There is no process to review measures and identify whether or not they are appropriate. That is a major factor affecting whether measures change in line with organisational circumstances’’). Within company 7 the leadership of the managing director was clearly the main driver of measurement activity. ‘‘The ability and energy of the managing director drive measures and measurement. He prompts other board members to review measures and ensure that they are relevant and appropriate to the business and reflect what is important.’’ The availability of management time to reflect on measures was also considered to be a major constraint. The group technical and quality director identified that: ‘‘In previous years we have had too many measures. We need to focus on fewer important objectives’’. He also noted that the frequency with which measures are reviewed is dependent on the availability of management time. Similarly the availability of management skills is also a key determinant of the ability to review and modify measures. This will affect when inappropriate measures are identified and the ability to change measures to make them appropriate’’. He identified the need for systems that could accommodate a hierarchy of measures, reporting the few important measures, but enabling analysis of the many underlying measures of the drivers of performance. Table II summarises the key factors that facilitate and inhibit the evolution of performance measurement systems in each of the case study companies. Evidence from the case study companies demonstrates the need for companies to change their performance measures as the organisation’s circumstances change. The group technical and quality director in company 7 pointed out: ‘‘If people don’t think measures are relevant they won’t use them, so they won’t evolve’’. This clearly demonstrates that in order for an organisation to have performance measures that evolve over time, they must
5
4
3
2
1
Company
Barriers to evolution Off the shelf systems insufficiently flexible Availability of skills to effectively collect and analyse data
Senior management inertia Measures not used to manage the business Time consuming and costly data collection
Management inertia Inflexible IT/finance systems Incompatibility of measures/inconsistent approach Culture – ad hoc measurement, no integrated approach or PM function Individual inertia/resistance to measurement Time wasted producing reports Ability to quantify performance Measures lacking credibility Measurement not used to manage the business (need new mind set) Accounting systems focus Inconsistent approach to measurement (due to changes in ownership and management) Lack of flexible systems to collect and analyse data (continued)
Facilitators of evolution
Senior management driving measurement activities Development of in-house IT systems Use of accepted communication media to communicate, generate feedback and involve all employees Integration of measurement with strategy development and review Consistent approach to measurement
New Web-based system developed In-house systems provide required flexibility Measurement included in business process review Alignment of rewards to measures Need for measures to evolve considered important Common understanding of objectives and the need to improve
Enthusiastic champion of measurement Contact with external research bodies to keep up to date with developments in measurement practices Make measurement a business issue – manage with measures
Enthusiastic champion to kick off ‘‘measurement revolution’’ The need for succession planning identified
Top level management support is critical User involvement in designing measures Alignment of rewards
Performance measurement systems 1233
Table II. Summary of case study findings
Facilitators of evolution
Dedicated PM resource (review of measures to ensure action is taken, IT and operational responsibility, credible sponsor) Integrated approach to measurement Open and honest process for reviewing measurement Centres of practice established to share knowledge Involvement of those being measured/local ownership of measures Measures linked to individual objectives Measurement not owned by finance Alignment of personal rewards Away day to review measures
Top management support for measurement The need for improved electronic reporting including hierarchy of measures and drill down facilities identified
6
7
Table II.
Company
1234 No review process in place Management time main constraint to reviewing measures Too many measures/lack of focus Incompatible measures – barrier to effective use Measures, actions and rewards not always aligned
Cross-functional ownership of measures/performance Staff/skill retention – lose of skills to analyse data and redefine measures Payroll and union systems Incompatible systems/measurement in different locations/ business units
Barriers to evolution
IJOPM 22,11
have a set of performance measures that is effectively used throughout the organisation. Companies 3 and 5 were prevented from maintaining a relevant set of measures by senior management, who continued to use financial measures to manage the business, despite the availability of a more balanced set of measures. Given that the availability and effective use of measures is a pre-requisite to their evolution, a key question that remains is how do companies know when they should change the measures they use. Each of the case study companies approached this problem in different ways, however each company also encountered considerable barriers to effective evolution of measures. As a result no organisation demonstrated a complete solution to the problem of managing the evolution of their measurement system. Analysis of the barriers that the case study organisations encountered, and approaches that different organisations used to overcome them, provide significant insight into the way that the evolution of measurement systems can be managed. Barriers to and facilitators of evolution The previous section presents the main findings of the case studies undertaken and Table II summarises the main barriers and facilitators of the evolution of performance measurement systems found within the companies studied. As presented, the findings provide an insight into the factors affecting the evolution of measures in each of the individual organisations. However, to draw generic lessons from the data collected it is necessary to identify common problems encountered by the organisations studied and solutions to these problems found in other organisations. Some of the organisations, notably companies 2 and 7, identified the need for management processes to ensure the review of performance measurement is prompted and that measures are changed as appropriate. Companies 1 and 2 overcame such problems by incorporating measurement into regular strategy and business process reviews. In company 1, considerable attention was focused on measurement as a key tool to support the achievement of strategic objectives. As a result measures were a fundamental part of the annual strategy review. Meanwhile in company 2, each business process had clearly defined performance measures. As a result, the process audits assessed the effectiveness of measurement and prompted remedial action as necessary. These comparisons suggest that absence of an effective process is a commonly encountered barrier to the evolution of measurement systems. Effective processes enable identification of changes in circumstances that necessitate changes in measures and ensure that measures are appropriate. Companies 4 and 6 both highlight the lack of the appropriate skills as a barrier to identifying the measures that need to be changed and to the modification of measures. Historically, high staff turnover had significantly affected the ability of company 6 to retain the skills required to analyse performance data, identify whether measures remain appropriate and design
Performance measurement systems 1235
IJOPM 22,11
1236
new measures when necessary. The availability of management time to reflect on measures was also found to be a constraint on evolution (companies 4 and 7). In order to overcome these issues, company 6 devoted dedicated resources to measurement, thereby giving individuals the responsibility for ensuring that measurement remained appropriate to the organisation’s circumstances. This dedicated resource acted as support for measurement activities, including use of measures, analysis of data and ensuring that measures reflected the requirements of users. The resource acted as a focal point for the development and maintenance of internal measurement skills including the development of appropriate information systems. The lack of the necessary skills and human resources (people) is the second barrier to evolution to be identified from the findings. The necessary skills include: the ability to identify when measures are no longer appropriate to measure the organisation’s performance; and the ability to refine measures to reflect the organisation’s new circumstances. The lack of flexibility of information systems, especially accounting systems, was considered to be a barrier to the evolution of measures by a number of companies (companies 1, 2, 5 and 6). Company 6 indicated that the implementation of an ERP system resulted in loss of functionality that had been developed to aid analysis of performance measurement data. Although there are many software products designed to support performance measurement on the market, company 1 in particular found ‘‘off the shelf systems’’ to be insufficiently flexible, limiting the ability to modify of measures. Companies 1, 2 and 6 identified the need to design data collection and reporting systems so that they facilitate the identification of inappropriate measurement and enable the change of data collection, analysis and reporting tools. In company 1 this was included in a strategic information system developed in-house. Company 2 developed a Web-based system that enabled consistent and flexible measurement on a global scale, while company 6 ensured that their dedicated measurement personnel had both operations and information systems responsibilities, ensuring that systems were developed to reflect the requirements of operational measurement. Inflexible systems are the third barrier to evolution to be drawn from the findings. The acceptance of measurement throughout the organisation was identified as a key prerequisite of evolution in each of the companies studied. This was linked to the importance placed on maintaining an effective and efficient measurement system, including the benefit derived from measurement activities in relation to the effort required. In each case, establishing a culture that embraced the use of performance measurement to manage the business was crucial. Without such a culture measurement was considered to be a nonvalue adding activity which was to be endured, rather than a tool to support business decision making. Company 1 used existing communication media to establish such a culture throughout the organisation. Use of statutory safety reports and briefings
ensured that all employees received business performance information, while mechanisms were put in place to encourage, collect and respond to queries from all employees regarding information reported. By demonstrating that this was an open and honest process, an average of 300 questions and suggestions per month were generated from a work force of approximately 600, demonstrating that measures were actually being understood and used. Similarly, management meetings in company 6 included open discussion of the relevance of performance measures and the way in which they could be modified and improved to increase their utility. In both cases, developing an open and honest culture in which measurement was used to support improvement rather than a tool to punish individuals was considered crucial to use maintenance of relevant performance measures. Thus the data suggests that culture that is inappropriate to the use (and change of measures) is the fourth key barrier that was identified. This further analysis of the data identifies four key themes commonly observed within the case study organisations. These are demonstrated through the examples discussed. Table III presents the findings from the case studies structured around four themes that emerge from the data collected. This demonstrates that these themes comprehensively cover all of the case study data as presented in Table II. Framework of factors affecting the evolution of performance measurement systems It is evident from the summary of the interviews shown in Table III that the evolution of measurement systems is a complex phenomenon to study. At the most fundamental level the research reported in this paper has identified that before a measurement system can evolve it has to be used actively (use). It is worth noting at this point that a performance measurement system itself consists of several components, including: . Individual measures that quantify the efficiency and effectiveness of actions. . A set of measures that combine to assess the performance of an organisation as a whole. . A supporting infrastructure that enables data to be acquired, collated, sorted, analysed, interpreted and disseminated (Neely, 1998). Assuming that these elements exist and the measurement system is actually used then evolution of the measurement system is possible. Typically, this starts with a trigger, which can be either internal or external in nature. Companies 1 and 5 both provide examples of an external trigger, with significant changes in their consumer markets following deregulation. These changes significantly affected the assumptions underpinning the companies’ objectives and competitive priorities, necessitating realignment of measures. In Company 6, monthly management meetings and an away day provide
Performance measurement systems 1237
Maintain PM capability (6) Dedicated PM resource (6) facilitation of use of measures (6) ensure action is taken (1, 2, 6) prompt review of measures (6) credible sponsor (1, 6, 7) IT and operational responsibilities (6) Skills/succession planning (4) Involvement of those being measured/local ownership of measures (6) Community of users of measures (6)
Develop in-house/customised IT systems (1, 2): flexible (1, 2) Web-based (2) electronic reporting (7) hierarchy of measures (7) linked to strategy deployment/business process review (1, 2) Maintain internal systems development capabilities (1, 2, 6) Integration of operations and IT (budgets, responsibility, etc.) (6)
People
Systems
Table III. Recategorised summary of case study findings
Integration of measurement with strategy development and review (company 1) Integration of measurement with business process review (2) PM ‘‘function’’ the focal point of measurement activity (6) Forum to discuss appropriateness of measures (6) Implementation of common definitions/metrics (3, 7) Consistent approach to measurement across all areas of the business (1) Away day to measures (6) Involvement of external bodies (3) User involvement in measurement (5)
(continued)
Inflexible legacy systems: data collection (4, 5) reporting (4, 5) Inflexible ERP systems – loss of functionality (6) Inappropriate ‘‘off the shelf’’ systems (1)
Lack of appropriate skills: to identify appropriate measures (4, 6, 7) to design measures/quantify performance (4) to collect accurate data (1) to analyse data (6) High staff turnover (6) Lack of management time (4, 7) Ownership of cross-functional (6)
Lack of proactive review process (7) Inconsistent approach to measurement: over time (5) between locations/business units (3, 6, 7) no integrated measurement function (3) Insufficient time to review measures: lack of management time (4, 7) too much data reported (4, 7) The need to trend measures limits ability change (7) Lack of data analysis (5, 6)
Barriers to evolution
1238
Process
Facilitators of evolution
IJOPM 22,11
Culture
The need for evolution considered to be important (2, 6, 7) Communication: use of accepted medium (1) feedback all actions (1) engage all employees (1) Measurement integrity is encouraged: open and honest discussion of performance (6) no blame culture (6) discouragement of ‘‘gaming behaviour’’ (6) Ongoing senior management support/champion for measurement (all companies): continued focus on measurement (1, 6) identify and remove barriers to use/change of measures (1, 6) Establish common understanding of objectives (2) Integration/alignment of reward systems (2) Measurement not owned by finance (6) Alignment of measures and rewards (2, 5, 6)
Facilitators of evolution
Senior management inertia (2, 3) Individual inertia/resistance to measurement (4) Ad hoc approach to measurement (3) Lack of alignment of actions with measures (7) In appropriate use of measures/measures not used to manage the business (2, 5) Rigid remuneration and union systems (6)
Barriers to evolution
Performance measurement systems 1239
Table III.
IJOPM 22,11
1240
examples of internal triggers which prompted review of the relevance of current measures given changes in circumstances. Other such triggers were also identified that prompted the realisation that measures were inappropriately designed for their purpose, that use of measures prompted inappropriate behaviour or that circumstances, such as competitive requirements, changed. Once the trigger has been received then the first stage in the evolution of the measurement system is to reflect on the performance measurement system and identify whether it remains appropriate given changing organisational circumstances. This stage of the evolutionary process is known as reflect (reflect) and the research identified several barriers that prevent it from occurring in organisations, most crucially those associated with process, people, infrastructure and culture: . Absence of an effective process. Company 7 highlighted the lack of an effective process as the main barrier to reflection, while in both companies 4 and 7 there was insufficient management time set aside to reflect on performance measures. . Lack of the necessary skills and human resources. Companies 1, 4, 6 and 7 each identified a lack of appropriate skills to analyse data and identify inappropriate measures. Company 6 specifically highlighted that high staff turnover caused problems in retaining people with the skills necessary to identify which measures are inappropriate. Company 4 also highlighted that the lack of succession planning was a barrier to reflection. . Inflexible systems. These were identified as barriers to reflection. In particular, company 6 found ERP system implementation led to lost analysis functionality required to investigate performance trends and causes of performance variances. . Inappropriate culture. Companies 4 and 6 both highlighted individuals who were resistant to reflection on and change to measures as they did not want measures to more effectively reflect specific dimensions of performance for which they were responsible. Lack of alignment of measures with rewards was also found to be a barrier to reflection in company 7. Alignment of measures with rewards ensures that those being measured have an incentive to reflect on measures and prompt their evolution. During the reflection stage, each of the constituent parts of the performance measurement system should be critically appraised and reviewed to ensure that they remain appropriate to the requirements of the organisation. Many tools and techniques have been developed to help organisations design performance measures and measurement systems. Several of these tools can be applied to reflect on the content of an organisation’s current performance measurement system. For example, the performance measurement record sheet (Neely et al., 1996) lists the characteristics of a performance measure, any of
which might be affected by changes in the organisation’s circumstances. Many of the performance measurement frameworks that have been proposed (Kennerley and Neely, 2000) might also support reflection on the relevance of the set of measures used by the organisation. Furthermore, tools such as the Performance Measurement Questionnaire (Dixon et al., 1990) are specifically designed to help an organisation to identify the appropriateness of their measurement system. Reflecting on the measurement system will enable required changes to be identified and will in turn trigger modifications (modify). In addition external triggers, such as changes in legislative or regulatory requirements, and/or changes in ownership can lead to the imposition of new performance measures, which will also prompt the modification stage. In turn the modification stage will result in changes to the constituent elements of the measurement system. Once these changes have been enacted then the modified measurement system can be said to have been deployed (deploy) and hence the cycle of evolution can start again. This entire evolutionary cycle is illustrated in Figure 2, which contains a framework of the factors affecting the evolution of measurement systems. The key to this discussion is to recognise that the case study data collected demonstrates that to manage effectively the evolution of performance measures, an organisation must consider several inter-related issues: (1) The active use of the performance measurement system is a prerequisite to any evolution. (2) The performance measurement system itself consists of three interrelated elements (individual measures, the set of measures and the
Performance measurement systems 1241
Figure 2. Framework of factors affecting the evolution of performance measurement systems
IJOPM 22,11
1242
enabling infrastructure). Each of these elements must be considered during the evolution of the performance measurement system. (3) There are four stages of evolution – use, reflect, modify and deploy. These form a continuous cycle. (4) Barriers exist that will prevent the evolutionary cycle from operating. These barriers can be overcome if the evolutionary cycle is underpinned by enabling factors – broadly categorised under the headings: people, process, people, infrastructure and culture. Specifically, a well designed measurement system will be accompanied by an explicitly designed evolutionary cycle with clear triggers and: . process – existence of a process for reviewing, modifying and deploying measures; . people – the availability of the required skills to use, reflect on, modify and deploy measures; . infrastructure – the availability of flexible systems that enable the collection, analysis and reporting of appropriate data; . culture – the existence of a measurement culture within the organisation ensuring that the value of measurement, and importance of maintaining relevant and appropriate measures, are appreciated. Discussion The literature and case study data presented clearly show first, the importance of managing measurement systems so that they change over time and second, the complex range of interrelated factors that affect the evolution of performance measurement systems. The literature highlights many of the issues affecting the management of change within organisations. This paper discusses many of these issues in the context of case study data relating to performance measurement system evolution. A considerable amount has been written about the design and implementation of measurement systems and a number writers have identified the need to reflect on measures to ensure that they remain relevant as the organisation changes. The research findings echo the themes identified in the literature concerning the external and internal drivers of change affecting organisations and the need for organisations to have effective process in place to identify these changes and when they necessitate changes to measurement systems. However, there is little discussion in the literature of what to do once that reflection has taken place. The data collected clearly show that the process of managing the evolution of measurement systems consists of a number of stages that have to date received little attention. In addition to reflection, consideration should be given to how measures are to be modified and how modified measures are to be deployed without embarking on a wholesale performance measurement system redesign project.
It is also clear that for measurement systems to evolve effectively there are key capabilities that an organisation must have in place (i.e. effective processes; appropriate skills and human resources; appropriate culture; and flexible systems). The research demonstrates how lessons from different strands of literature such as the need for the appropriate resources (Greiner, 1996) and capabilities (Gabris, 1986); the appropriate culture (Tichy, 1983); willingness to change (Kotter, 1996); and relevant processes (Bourne et al., 2000; Bititci et al., 2000) can be drawn together into a structured framework. The data indicates that organisations should consider these capabilities at each stage of the evolutionary cycle, as they are fundamental to effective evolution. However, little consideration is given to these capabilities in the literature concerning the design and implementation of measurement systems. It is the development and maintenance of these capabilities within an organisation that will determine whether its measurement systems evolve effectively. As such, reviewing the availability of these capabilities is an important stage in the management of measurement systems over time. This reflects the need to review and update measurement systems at three different levels, i.e. the individual measure; the set of measures; and the supporting infrastructure, and shows that these capabilities are an integral part of that supporting infrastructure. The framework presented provides a structured view of the factors affecting the evolution of performance measures and measurement systems. It conceptualises a very complex combination of factors affecting the evolution of measurement systems into a manageable form. Conclusions Although the issue of development of effective performance measures has received considerable attention from both academic and practitioner communities, neither has satisfactorily addressed the issue of how performance measures should evolve over time in order to remain relevant. The research reported in this paper provides an understanding of how measurement systems can be managed so that a dynamic and relevant set of performance measures can be maintained, reflecting an organisation’s changing requirements. It provides an understanding of the factors, both internal and external to the organisation, that facilitate and inhibit the introduction of new measures, the modification of existing measures and deletion of obsolete measures. These factors are presented in a framework that illustrates the process, people, infrastructure and culture capabilities that an organisation must demonstrate in order to manage the evolution of measures. The paper discusses many issues of relevance to the growing literature in the field of performance measurement while providing organisations with a practical tool to help them establish an effective performance measurement system. Ensuring that evolution of measurement systems is effectively managed over time is important if another measurement crisis and revolution is to be avoided.
Performance measurement systems 1243
IJOPM 22,11
1244
References Bititci, U.S., Turner, T. and Begemann, C. (2000), ‘‘Dynamics of performance measurement systems’’, International Journal of Operations & Production Management, Vol. 20 No. 6, pp. 692-704. Bourne, M., Neely, A., Mills, J. and Platts, K. (1999), ‘‘Performance measurement system implementation: an investigation of failures’’, Proceedings of the 6th International Conference of The European Operations Management Association, Venice, 7-8 June, pp. 749-56. Bourne, M., Mills, J., Wilcox, M., Neely, A. and Platts, K. (2000), ‘‘Designing, implementing and updating performance measurement systems’’, International Journal of Operations & Production Management, Vol. 20 No. 7, pp. 754-71. Bruns, W. (1998), ‘‘Profit as a performance measure: powerful concept, insufficient measure’’, Performance Measurement – Theory and Practice: The First International Conference on Performance Measurement, Cambridge, 14-17, July. Dixon, J.R., Nanni, A.J. and Vollmann, T.E. (1990), The New Performance Challenge – Measuring Operations for World-Class Competition, Dow Jones-Irwin, Homewood, IL. Eccles, R.G. (1991), ‘‘The performance measurement manifesto’’, Harvard Business Review, January-February, pp. 131-7. Fitzgerald, L., Johnston, R., Brignall, T.J., Silvestro, R. and Voss, C. (1991), Performance Measurement in Service Businesses, The Chartered Institute of Management Accountants, London. Frigo, M.L. and Krumwiede, K.R. (1999), ‘‘Balanced scorecards: a rising trend in strategic performance measurement’’, Journal of Strategic Performance Measurement, Vol. 3 No. 1, pp. 42-4. Gabris, G.T. (1986), ‘‘Recognizing management techniques dysfunctions: how management tools often create more problems than they solve’’, in Halachmi, A. and Holzer, M. (Eds), Competent Government: Theory and Practice, Chatelaine Press, Burk, VA, pp. 3-19. Ghalayini, A.M. and Noble, J.S. (1996), ‘‘The changing basis of performance measurement’’, International Journal of Operations & Production Management, Vol. 16 No. 8, pp. 63-80. Globerson, S. (1985), ‘‘Issues in developing a performance criteria system for an organisation’’, International Journal of Production Research, Vol. 23 No. 4, pp. 639-46. Greiner, J. (1996), ‘‘Positioning performance measurement for the twenty-first century’’, in Halachmi, A. and Bouckaert, G. (Eds), Organizational Performance and Measurement in the Public Sector, Quorum Books, London, pp. 11-50. Johnson, H.T. (1983), ‘‘The search for gain in markets and firms: a review of the historical emergence of management accounting systems’’, Accounting, Organizations and Society, Vol. 2 No. 3, pp. 139-46. Johnson, H.T. and Kaplan, R.S. (1987), Relevance Lost – The Rise and Fall of Management Accounting, Harvard Business School Press, Boston, MA. Kaplan, R.S. (1984), ‘‘The evolution of management accounting’’, The Accounting Review, Vol. 59 No. 3, pp. 390-418. Kaplan, R.S. and Norton, D.P. (1992), ‘‘The balanced scorecard – measures that drive performance’’, Harvard Business Review, January/February, pp. 71-9. Kaplan, R.S. and Norton, D.P. (1993), ‘‘Putting the balanced scorecard to work’’, Harvard Business Review, September/October, pp. 134-47. Keegan, D.P., Eiler, R.G. and Jones, C.R. (1989), ‘‘Are your performance measures obsolete?’’, Management Accounting (US), Vol. 70 No. 12, pp. 45-50.
Kennerley, M.P. and Neely, A.D. (2000), ‘‘Performance measurement frameworks – a review’’, Proceedings of the 2nd International Conference on Performance Measurement, Cambridge, pp. 291-8. Kotter, J.P. (1996), Leading Change, Harvard Business School Press, Boston, MA. Lynch, R.L. and Cross, K.F. (1991), Measure Up – The Essential Guide to Measuring Business Performance, Mandarin, London. Maskell, B. (1989), ‘‘Performance measures for world class manufacturing’’, Management Accounting (UK), May, pp. 32-3. Meyer, M.W. and Gupta, V. (1994), ‘‘The performance paradox’’, in Straw, B.M. and Cummings, L.L. (Eds), Research in Organizaional Behaviour, Vol. 16, JAI Press, Greenwich, CT, pp. 309-69. Neely, A. (1998), Measuring Business Performance – Why, What and How, Economist Books, London. Neely, A.D. (1999), ‘‘The performance measurement revolution: why now and where next’’, International Journal of Operations and Production Management, Vol. 19 No. 2, pp. 205-28. Neely, A.D., Kennerley, M.P. and Adams, C.A. (2000), The New Measurement Crisis: The Performance Prism as a Solution, Cranfield School of Management, Cranfield. Neely, A.D., Mills, J.F., Gregory, M.J., Richards, A.H., Platts, K.W. and Bourne, M.C.S. (1996), Getting the Measure of Your Business, Findlay Publications, Horton Kirby. Pettigrew, A. and Whipp, R. (1991), Managing Change for Competitive Success, Blackwell, Oxford. Scott, W.R. (1995), Institutions and Organizations: Theory and Research, Sage Publications, London. Senge, P.N. (1992), The Fifth Discipline: The Art and Practice of the Learning Organization, Century Business Press, London. Tichy, N.M. (1983), Managing Strategic Change: Technical, Political, and Cultural Dynamics, John Wiley & Sons, New York, NY. Townley, B. and Cooper, D. (1998), ‘‘Performance measures: rationalization and resistance’’, Proceedings of Performance Measurement – Theory and Practice: the First International Conference on Performance Measurement, Cambridge, 14-17, July, pp. 238-46. Waggoner, D.B., Neely, A.D. and Kennerley, M.P. (1999), ‘‘The forces that shape organisational performance measurement systems: an interdisciplinary review’’, International Journal of Production Economics, Vol. 60-61, pp. 53-60. Wisner, J.D. and Fawcett, S.E. (1991), ‘‘Linking firm strategy to operating decisions through performance measurement’’, Production and Inventory Management Journal, Third Quarter, pp. 5-11.
Performance measurement systems 1245
The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/0144-3577.htm
IJOPM 22,11
1246
Adding value to performance measurement by using system dynamics and multicriteria analysis Se´rgio P. Santos Faculty of Economics, University of Algarve, Faro, Portugal
Valerie Belton and Susan Howick Department of Management Science, University of Strathclyde, Glasgow, UK Keywords Performance measurement, Management, Systems design, Integration Abstract The design, implementation and use of adequate performance measurement and management frameworks can play an important role if organisations are to succeed in an increasingly complex, interdependent and changing world. Yet, despite widespread recognition of the importance of performance assessment, there are some issues that require further study if measurement systems are to be effective in the process of management through monitoring and decision making. This article proposes and seeks to illustrate that the use of system dynamics and multicriteria decision analysis, as part of a broader framework for performance measurement, can address some of the issues identified. The use of these approaches, independently and in an integrated manner, provides a means of exploring the dynamic complexity of organisations, making explicit trade-offs between performance measures, and assessing the impacts of initiatives to improve performance, thereby enhancing understanding and ultimately contributing to the improvement of organisational performance.
1. Introduction The environment within which most organisations operate is changing rapidly. Organisations failing to adapt and respond to the complexity of the new environment tend to experience, sooner or later, survival problems. In this climate of change, the development, implementation and use of adequate performance measurement and management frameworks is one of the major challenges confronting organisations and can contribute significantly to their success. Over the past decade, several performance measurement frameworks, which seek to address some of the criticisms of traditional accounting based measurement systems and to deal with a rapidly changing environment, have been proposed. The balanced scorecard (Kaplan and Norton, 1992), the performance pyramid (Lynch and Cross, 1991), the results and determinants framework (Fitzgerald et al., 1991), and more recently, the performance prism (Neely et al., 2002), are among the better known examples. International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, pp. 1246-1272. # MCB UP Limited, 0144-3577 DOI 10.1108/01443570210450284
Financial support for the research being carried out by Se´rgio P. Santos has been gratefully received from the Fundac¸a˜o para a Cieˆncia e a Tecnologia, under grant SFRH/BD/823/2000.
However, in spite of the availability of a variety of approaches, it is Adding value to recognised that there are issues that deserve further research if performance performance measurement systems are to achieve their full potential to improve measurement organisational performance through the provision of support to the associated monitoring and decision-making processes. In this paper we propose a framework to assist performance measurement 1247 and management incorporating the integrated use of multicriteria decision analysis (MCDA) and system dynamics (SD), which provides a means of addressing some of the issues that have been raised. From a research perspective the proposal should be viewed as part of a continuous learning cycle as suggested by Kolb (1984), seen in Figure 1. The authors have extensive concrete experience in the use of MCDA (Belton, 1985, 1993; Belton and Stewart, 2002) and SD (Howick and Eden, 2001) to develop understanding and to inform decision making in a wide range of organisational contexts, including performance measurement. Much of this work has been conducted as action research (Coughlan and Coghlan, 2002; Eden and Huxham, 1996; Meredith et al., 1989) and reflection on this experience led the authors to the proposal, initially as an abstract concept, that there is potential synergy to be derived from the integrated use of MCDA and SD, particularly in the context of performance measurement and management. Our aim was to develop this concept, initially informed by literature review and conceptual experimentation alongside previous practical experience, to a level at which it represented a practically feasible and potentially valuable methodology. This has brought us to the framework proposed in this paper, which is illustrated by a realistic (i.e. based on experience and knowledge of the specific context) but hypothetical (in that the detail and data used do not correspond to a specific organisation) application in the management of health care resources. The testing of this idea in the practical context is the next stage of the research, currently in its initial phase, which will take the form of one or more action research studies. Benefits of the use of MCDA and SD to support decision making in organisations derive as much from the process as the outcome of analysis, and an important aspect of this is the learning, which results from an effectively facilitated intervention (Belton and Elder, 1994; Belton and Stewart, 2002; Morecroft and Sterman, 1994). We believe that the
Figure 1. Kolb’s experiential learning cycle
IJOPM 22,11
1248
incorporation and integration of these two approaches in the context of performance measurement and management will further enhance shared understanding of the issues and complexities faced by an organisation, in particular the appreciation of the nature and implication of the dynamics of the system and of the performance trade-offs which may need to be confronted. It is not our intention to suggest a methodology in competition to existing and welldeveloped approaches, but to demonstrate how MCDA and SD, individually or in combination, could enhance particular aspects of any performance measurement framework. Neither do we wish to claim that the proposed framework is a panacea for all ills befalling performance measurement, simply that it seeks to address some of the issues to be highlighted in section 2 of the paper. The paper is structured as follows. As indicated in section 2, we discuss why performance measurement systems frequently fail in supporting the decisionmaking process, focusing particularly on issues where SD and MCDA can be helpful. In section 3 we outline the strengths of these two approaches to show how their use can bring new insights to inform and support performance measurement and management, illustrating this discussion by an example application from the health care sector. In section 4 we conclude with some closing remarks and an outline of the ongoing research agenda. 2. Performance measurement – some emerging issues Any evaluative review of the literature illustrates that performance measurement is a field attracting considerable attention and where remarkable progress has been made over recent years. The theoretical development of better integrated, balanced and strategically driven performance measurement frameworks has been accompanied by an increase in the practice of performance measurement. However, in spite of the significant advances made, there are still some issues which require further study if measurement systems are to be fully effective in the process of management through monitoring and decision making. Diverse reasons may be offered to justify why many efforts to improve performance through the design and implementation of measurement systems have not met with great success. Taking a holistic view of the field, these reasons can be grouped into two major intertwined categories: the first refers to the design and implementation of performance measurement systems; the second refers to the analysis and use of the information resulting from measurement. It is widely recognised that poorly designed performance measurement systems can seriously inhibit their implementation and, consequently, their ultimate impact. It is vital therefore that organisations fully realise the importance of developing appropriate measurement frameworks. A review of the literature suggests that although significant progress has been made in the last decade in developing integrated, balanced and strategically driven performance measurement systems (for a comprehensive review of literature
on performance measurement system design the reader is referred to Neely et Adding value to al., 1995), there are some issues that have not yet been fully addressed. performance Processes and procedures have been suggested to identify performance measurement measures and to group them in a fashion that makes them easier to interpret and understand. Several authors have recognised however that much more has to be done to identify the relationships between measures (see, for example, 1249 Neely, 1999; Flapper et al., 1996; Bititci et al., 2000). In reality, in spite of the recognised importance of understanding the relationships among the various performance measures in developing a comprehensive measurement system (Eccles and Pyburn, 1992), far too many organisations still define their measurement systems without understanding the dynamic interdependencies and trade-offs between measures and ultimately the process underlying performance generation. Both the identification of appropriate measures and the explicit consideration of trade-offs between them can be significantly assisted if the relationships among measures are mapped and understood. However, it is curious to note that with a few exceptions, little consideration is given in the literature to the relationships between performance measures. Trying to identify factors affecting performance and to explicitly represent their relationships, Suwignjo et al. (2000) suggested the use of cognitive maps and Kaplan and Norton (2001) the use of strategy maps. These cause and effect diagrams are very valuable to capture and make explicit the managers’ ‘‘theory of the business’’ and consequently they may prove very useful in identifying appropriate performance measures. Nevertheless, strategic maps and/or cognitive maps do not allow participants to understand fully the implications of interconnections between the factors affecting performance due to the existence of non-linear interactions, delays, feedback loops and other elements that give rise to dynamic complexity and which are not incorporated in these types of mapping. To deal with the dynamic complexity inherent in social systems and to infer dynamic behaviour, quantitative simulation is required (Senge, 1990; Sterman, 1989a, b). Therefore, and particularly in those situations where it is important to understand the interactions among the measures over time, the value added by strategy maps and/or cognitive maps can be significantly increased if they are complemented with simulation modelling. The combined use of qualitative and quantitative modelling enriches the analysis and can provide very useful insights for the design of measurement systems. Once the performance measurement system has been designed, it has to be implemented. There is, however, some evidence that even in those cases where measurement systems have been successfully designed their implementation is not always a straightforward task. Neely et al. (2000) argue that this is mainly due to fear, politics and subversion. Dumond (1994) suggests that the problems in the implementation of measurement systems are mainly due to lack of communication and dissemination of results. Although there is no ‘‘right’’ solution to these problems it is widely accepted that an active leadership
IJOPM 22,11
1250
involvement and an effective communication are key factors to a successful implementation of performance measurement systems and to an effective use of the performance data. It is also important to make clear in the early stages of the process that the objective of measurement is not to punish or blame individuals but rather to provide information that helps the organisation to take appropriate action with the ultimate goal of improving operations. Unless appropriate organisational responses to performance results are carried out, performance measurement efforts may prove worthless. Both the implementation of measurement systems and using them to manage organisational performance appear, nevertheless, to be areas in which progress has been limited to date (Neely, 1999; Bourne et al., 2000). However, an effective use of the performance information to improve operations is a complex undertaking which may prove difficult without the support of appropriate tools. On the one hand, understanding the real causes of poor performance and determining the proper action plan for performance improvement requires, in most cases, detailed analysis of the structure of the problem under study and the consideration of trade-offs. On the other hand, due to the limited information processing capabilities of the human brain, decision makers lack the ability to process effectively all the information necessary to develop and implement more coherent and better-informed action plans. Therefore, the use of analytical tools to help managers increase their understanding about the reasons why a particular level of performance is observed should be encouraged. According to Lebas (1995) only when we understand which of the steps in the performance generation process is problematic, can effective corrective action be designed. Far too often, organisations fail to analyse the performance data properly and end up undertaking symptomatic interventions. Because these interventions are based on symptoms rather than on the underlying causes of problems, often they are not the most effective in achieving lasting performance improvements. Many times the actions undertaken to solve or improve a problem actually make it worse, or create another, unforeseen, problem. Even when a detailed analysis of the structure of the problem is carried out, the identification and evaluation of appropriate corrective actions is not always a straightforward task. Frequently, the achievement of high levels of performance on one measure can only be obtained at the expense of performance on one or more other measures, making trade-offs among these measures inevitable. Skinner (1974) in his classic paper on the focused factory, has long recognised that firms cannot excel in all aspects of performance, highlighting the need for explicit decisions about significant trade-offs. Further support for the need for trade-offs between different performance areas has been provided by a number of authors (see, for example, Banks and Wheelwright, 1979; Fine and Hax, 1985; Richardson et al., 1985; Eccles and Pyburn, 1992; Mapes et al., 1997; da Silveira and Slack, 2001). Despite the recognised importance of explicitly dealing with priorities and the relationships between performance measures (Neely et al., 2000), very little work has been
done to establish the nature of the trade-offs between these measures (Mapes et Adding value to al., 1997). The need for additional research dealing with the trade-offs among performance performance measures is, indeed, recognised in the literature by other authors measurement (see, for example, Ittner and Larcker, 1998). In summary, an evaluative review of the literature leads to two major conclusions. First, effectively measuring and managing of organisational 1251 performance is a complex and difficult task. Second, if performance measurement is to lead to enduring and continuous performance improvement, then the different stages of the performance measurement and management process (design of measurement systems, their implementation, analysis and use) must be successfully completed and form a continuous loop (Figure 2). It is also important to note that this process should be iterative and not a linear sequence of steps, as indicated by the arrows in the centre of the diagram. In the next section we will begin with a brief overview of SD and MCDA and we will then go on to discuss in greater detail how their use can bring new insights to inform and support performance measurement and management. In particular, we will discuss how these approaches can support the design, implementation, analysis and use of measurement systems, helping decision makers to close the loop diagrammed in Figure 2. 3. Adding value to performance measurement 3.1 System dynamics System dynamics was conceived and developed in the late 1950s and early 1960s at the Massachusetts Institute of Technology by Jay Forrester. Indeed, the advent of SD is generally considered to be the publication of Forrester’s pioneering book, Industrial Dynamics in 1961. Since then, significant advances have been made, and a cursory examination of the literature indicates that the number of organisations using SD models for the development of both strategic and operational policies is growing rapidly. An overview of SD can be found, for example, in Forrester (1961), Richardson and Pugh (1981) and Sterman (2000). SD models are frequently developed and used to represent, analyse, and explain the dynamics of complex systems. The dynamics or behaviour of a
Figure 2. The performance management process
IJOPM 22,11
1252
system is defined by its structure and the interactions of its parts. The main goal of SD is to understand through the use of qualitative and quantitative models how this behaviour is produced, and to use this understanding to predict the consequences over time of policy changes to the system. Although SD models can help decision makers in enhancing understanding of system behaviour over time, SD models do not concern themselves with the explicit evaluation of this behaviour. That is, a pattern of behaviour is frequently presented as preferable to another, based only on the modeller’s intuition (Gardiner and Ford, 1980). Some effort has been devoted since the early 1980s to the study of optimisation of system dynamic models (see, for example, Coyle, 1985). Despite this, it is recognised that evaluating, and choosing between alternative courses of action is not a straightforward task. In the context of SD, as in many others, the decision maker is confronted with a large and complex amount of information, usually of a conflicting nature and reflecting multiple interests. Consequently, the use of an appropriate MCDA approach can be very valuable to assist decision makers in organising such information in order to identify a preferred course of action (see, for example, Belton, 1985). 3.2 Multiple criteria decision analysis MCDA is now 30 years old, and it is an important area of operations research/ management science. Since the first session devoted to multicriteria analysis in a scientific congress, organised by Roy during the 7th Mathematical Programming Symposium, which was held in The Hague in 1969, the field of MCDA has seen remarkable growth. On one hand, important theoretical results have been achieved leading to the development of several multicriteria methods. On the other hand, the number of real world applications documented in the literature is increasing considerably. A synthesis of the main streams of thought in this field can be found in Belton and Stewart (2002). MCDA is designed to take explicit account of multiple and usually conflicting objectives in supporting the decision process. MCDA is an umbrella term which includes a range of different approaches sharing this common objective; it is common to group these approaches in three schools – aggregation models, outranking models and reference point models (Belton and Stewart, 2002). The approach, which will be described in this paper, falls into the school of aggregation models, but its application is informed by knowledge of the other approaches (Belton, 1990) and is guided by a constructivist (rather than a normative or prescriptive) philosophy. We adopt the view that, although some values will be well formed, on the whole preferences are not predetermined and the role of analysis is to help decision makers construct these through increasing understanding of the issue. In this way, MCDA methodologies can help decision makers to learn about the problems they face, about their own priorities and those of other stakeholders, and consequently to make better informed and justifiable choices, a view shared by many
prominent researchers in the field (see, for example, French, 1988; Goodwin and Adding value to Wright, 1998; von Winterfeldt and Edwards, 1986). performance Thus, a principal benefit of both SD and MCDA is seen to be the greater measurement understanding of the system of interest that can result from their use. However, for this to be achieved it is important that the process is facilitated by one or more persons with expertise in managing both the social and technical aspects 1253 of an intervention (Eden, 1990). 3.3 Using SD and MCDA to support the performance measurement and management process In this section we discuss how SD and MCDA can assist and add value to each one of the stages in the life cycle of the performance measurement and management process diagrammed in Figure 2. A simple hypothetical but realistic application in the health care sector is used to illustrate this process. These approaches have potential to be applied in a wide variety of organisations. However, three main reasons suggest that they can provide very valuable insights in supporting NHS hospital trusts in measuring and managing their performance, hence the choice of an illustrative example in this sector. First, hospital trusts, like many organisations, are complex systems in which several parts interact over time. For example, within a hospital the accident and emergency (A&E) department does not exist in isolation. It cannot operate independently of surgery, nor of the management of elective patients, nor of the management and staffing of beds on wards. Actions in any one part of the system affect the others (sometimes counter-intuitively), and consequently affect the dynamics of the whole system. As we will discuss in more detail below, for effective measurement and management of the performance of hospital trusts to take place it is vital that these interactions are understood and effectively managed. Second, the design of any system to measure and manage performance in hospital trusts needs to take into account the interests of multiple stakeholders including patients, health care professionals (clinical and nursing staff), administrative and managerial staff, purchasers (GPs fundholders and health authorities), as well as central government (NHS Executive). The large variety of points of view – either conflicting or overlapping – between these stakeholders demands that trade-offs between objectives are explicitly considered when assessing performance. Finally, health care systems around the world, including hospital trusts, are under pressure to measure their performance. However, in spite of the availability of several performance measurement systems, with different characteristics and of varying complexity, it is recognised that some of the issues discussed in section 2, which make performance measurement initiatives fall short of their potential, are common across these systems (see, for example, Ballantine et al., 1998). Therefore, attempts should be made to help hospital trusts to address these weaknesses.
IJOPM 22,11
1254
As already mentioned, the example that follows, while realistic, is only illustrative and the data do not relate to any real organisation. The apparent simplicity should not be interpreted as indicative of a lack of realism as the nature of modelling in practice is such that fitness for purpose is more important than completeness and a simple representation may be the most appropriate way of capturing a complex situation. Here the main purpose of the model is to illustrate the potential of and to provide meaningful insights into how SD and MCDA can be applied, as well as highlighting both advantages and limitations of their use. 3.3.1 Design. It is widely accepted that effective performance measurement systems should provide decision makers with information about the degree to which organisational objectives are achieved and how well an organisation is performing its tasks. To get this information, an appropriate set of performance measures is required. However, the issue of which performance measures a given organisation should adopt is not a straightforward one. Although the design of performance measures has been widely discussed in the literature (for an exhaustive review of the literature, see Neely et al., 1997), there is no consensus concerning the best way to develop performance measures. It is however recognised that performance measures should align with the strategic objectives of the organisation (see, for example, Globerson, 1985; Kaplan and Norton, 1992; Lynch and Cross, 1991). In this way, it is assured that the system will provide information on whether or not these strategic objectives are being successfully achieved. Additionally, it is assured that if corrective actions are required, steps consistent with these objectives will be adopted. Yet, despite this, it is recognised that several organisations develop performance measures on an ad hoc basis and without taking into consideration the relationships between measures. It is also recognised that even when more structured frameworks for performance measurement are adopted, little guidance on how the appropriate measures can be identified (Neely et al., 2000) and how to capture a holistic view of the system being assessed (Sloper et al., 1999), is provided. A good understanding of the relationships between measures offers many advantages. From one side, it can significantly assist the identification of appropriate performance measures. Lebas (1995), for example, suggests that understanding the processes underlying performance generation is the only way to define the measures that lead to appropriate actions. From another side, ignoring the relationships between performance measures and looking only to local performance measures may be a misleading guide to policy and to the overall performance of an organisation. Focusing on such measures, which relate to only a small part of a system (Fry and Cox, 1989) may result in suboptimisation of results (Harrington, 1991; Flapper et al., 1996). Because of the breadth of activities within organisations like hospital trusts where different activities are highly interconnected and where the influences from the external environment play an important role on their performance, capturing a holistic view of the system and understanding the relationships
among measures may be vital in developing a comprehensive and effective Adding value to performance measurement framework. performance As already mentioned, health care systems around the world have been measurement under pressure to measure their performance. In the UK the White Paper The New NHS: Modern, Dependable (Department of Health, 1997) emphasises the importance the government places on the measurement and improvement of 1255 health service quality, efficiency and effectiveness. To carry out these activities in the most successful way, the performance assessment framework (NHS Executive, 1999), based on the balanced scorecard, was published in April 1999 and has been continuously refined since then. The performance assessment framework for NHS hospital trusts comprises four areas of performance which, taken together, are intended to give a balanced view of the performance of the trusts: clinical effectiveness and outcomes; efficiency; patient/carer experience; and capacity and capability. Several performance indicators have been proposed to populate each one of these areas. Emergency re-admissions to hospital within 28 days of discharges as a percentage of live discharges; deaths within 30 days of surgery per 100,000 patients; actual length of stay compared to the average length of stay; and percentage of patients waiting less than six months for an inpatient admission are only a few examples (for a detailed list of performance indicators the reader is referred to Department of Health, 2002). It is important to emphasise, however, that although the list of performance indicators proposed by the Department of Health allows health professionals and NHS managers to know how the hospital trust compares with the defined targets on a range of measures, and therefore to know what is working well and what is not, this does not provide a strong basis from which to manage effectively for improvement. A list of performance indicators neither provides information about why the hospital trust is performing in that way, nor provides significant insights about possible intervention or leverage points. Furthermore, the lack of understanding of the interactions between measures prompts piecemeal solutions which can ultimately do more harm than good. Hospital trusts, like most organisations, are complex systems in which several parts interact over time. Unless some attempt is made to understand these interactions and to co-ordinate the actions of the different parts, it is very likely that hospital trusts will under-perform and that the performance measurement systems will be a misleading guide to policy. Several tools or facilitative processes can be used to foster creative thinking in order to identify performance measures and to increase understanding about their relationships. As stressed in the previous section, the use of cognitive maps and strategy maps has proved to be very valuable at this stage. However, these maps provide only a qualitative representation of the feedback structure of systems and, consequently, are not themselves very useful in inferring dynamic behaviour. To derive dynamic insights from complex feedback systems, quantitative simulation is required. An important step in conceptualising a system for quantitative SD model-building is to represent its
IJOPM 22,11
1256
Figure 3. Causal loop diagram
feedback structure in a visual way through causal loop diagrams (CLDs). CLDs are an important tool for identifying and representing feedback loops and given that it is the interaction between these loops that determines the dynamics of the system, they may prove very effective in helping to identify appropriate performance measures and to understand their dynamic relationships. CLDs, sometimes referred to as influence diagrams, are maps showing the causal relationships among a set of variables operating in a system and they are developed following well-established guidelines (see, for example, Coyle, 1996; Sterman, 2000). The CLD in Figure 3 is based on a model originally developed by Coyle (1984) and despite its simplicity it illustrates some of the advantages that can emerge from the use of this type of diagram. First, it gives a clear picture of the different elements of the problem and the interconnectedness between them. Cause and effect, feedback loops, delays and other elements that give rise to dynamic behaviour can be easily represented using CLDs. In the performance assessment framework the performance measure ‘‘waiting time’’ is considered a proxy measure of the patient/carer experience with hospital trusts. Lower values reflect a better experience of patients and consequently a higher satisfaction level. Figure 3 shows how this measure interacts with other ones. For example, an increase in the admission rate will tend to lead to a decrease in the time that patients have to wait until they are seen by a doctor. Conversely, an increase in the size of the waiting list should result in an increase in the waiting time, other factors remaining constant. The negative (or balancing) loops formed by these variables act to control the time that patients have to wait by controlling admissions. Therefore, taking a holistic view of the system being assessed through the development of qualitative diagrams and, as we shall illustrate later, of SD simulation models, teams tasked with developing measurement systems can better understand how the different measures interact with each other over time and in this way gain a greater (and shared)
understanding of the processes that determine the organisation’s performance. Adding value to For example, developing these models it becomes clear that performance performance measures, like length of stay in hospital and emergency re-admissions rates, measurement are linked measures and therefore they must be jointly assessed and managed. It becomes also clear that an effective co-ordination between GPs’ referrals and admissions to hospital is essential to achieve good levels of performance in 1257 measures such as waiting lists or waiting time. Second, as can be inferred from above, the use of CLDs allows the identification of intervention points or policy levers that can be used to control the performance of hospital trusts. For example, to increase the likelihood of achieving the desired level for the performance measure waiting time, a possible course of action is to reduce the duration of treatment. As can be seen in Figure 3, reducing the length of stay of patients in hospital would lead to an increase in the discharge rate and to a decrease in the number of patients in hospital, allowing higher admission rates and, ultimately, shorter average waiting times. However, Figure 3 also shows that while shortening the duration of treatment might lead to shorter waiting times, it also increases the likelihood of inappropriate discharges. Therefore, CLDs are also an effective tool in exposing trade-offs between performance measures. In this case, for example, the improvement of the efficiency of the hospital trust would be at the expense of the clinical effectiveness and outcomes of treatment. Furthermore, if it is reasonable to suppose that hospital trusts may in part control the performance of certain variables, there are others of which overall control is beyond them. The CLD shows that many factors, some of which are outside the direct control of hospitals, such as GP referrals or the capacity in community care, also have an important bearing on the performance achieved by a particular hospital. Finally, but not least, the process of building the CLD representation of the system, guided and assisted by an effective facilitator, helps to clarify people’s thinking on the subject and on their objectives. Developing SD models usually requires a significant amount of discussion among the members of the working group. This leads to a clearer understanding of the problem situation and fosters consensus and commitment about what should be measured, why and how. It is important to note, however, that while CLDs (qualitative SD) may help to visualise interdependencies and feedback processes that generate dynamic behaviour in the system being assessed, they do not allow rigorous conclusions to be drawn regarding this behaviour. For example, CLDs are unable to capture the stock (e.g. patients in waiting list) and flow (e.g. admission rate) structure of systems. Knowledge of the stocks and flows, along with feedback, are essential to understand the dynamics of a system. Furthermore, to fully understand the dynamics of complex systems quantitative models are essential. Only through the development and use of SD simulation models can managers fully understand the implications of non-linearity, feedback and delay among the performance measures and be able to identify the highest leverage points in the system. Moreover, using SD simulation models several alternative actions can
IJOPM 22,11
1258
be explored and their impact on the performance of the system tested in a riskfree environment. For example, as we will discuss in further detail in section 3.3.4, alternative plans or schemes for the allocation of resources between the many stages of the patient flow process can be tested using these simulation models. It is, however, important to note that for this understanding and learning to occur it is essential that the process is effectively facilitated (Richardson and Andersen, 1995; Vennix, 1996; Morecroft and Sterman, 1994) and that the problems and limitations of SD (Meadows, 1980; Legasto and Maciariello, 1980) are kept in mind and dealt with accordingly. The research carried out in the field suggests, for example, that to benefit fully from the SD models one should either be actively involved in the model building process or have a sufficient knowledge of the SD approach. Although significant progress has been achieved in recent years in developing tools that make the communication of insights generated by SD models easier and more effective, those requirements may constitute an obstacle to usage and implementation of SD models to those not directly involved in the modelling process. 3.3.2 Measurement. Having identified the performance dimensions and measures which the decision makers consider to be relevant in evaluating organisational performance, the next step is to set targets and to find out how well the organisation is achieving them. These targets can be set in different ways. One possibility is to establish targets based on a range of ‘‘acceptable performance’’. The upper limits of this range may include, for example, industry benchmarks and the lower limits may represent the lowest tolerable performance for each measure. Alternatively, a scale which ‘‘allows’’ an organisation to perform outside these limits may be used. As we will discuss in a moment, setting targets in this way allows not only to assess how well the organisation is performing against each individual measure but also to have an overall view of its performance by aggregating these measures into a few dimensions of performance or into a single indicator of overall performance. A single or a few key indicators of overall performance are valuable for several reasons. As suggested by Pritchard et al. (1990/1991), people find composite indicators of performance to be motivating given that these indicators allow them to see the results of their efforts on the overall performance of the organisation. In addition, a single or a few indicators of overall performance allow management teams to see if particular changes in the workplace either help or hinder overall performance. Finally, most people will judge the overall performance of the organisation even when composite indicators are not provided, and as Pritchard et al. (1990/1991) emphasise, this type of idiosyncratic judgement is usually much less accurate than designing overall performance indicators based on structured approaches. Several organisations have shown interest in designing overall performance indicators. Hospital trusts have been no exception. It is important to note that while the focus of the performance assessment framework is on individual measures, with the introduction, first, of the traffic lights performance system (Department of Health, 2000), and later of the performance ratings system
(Department of Health, 2001) the government shows a clear intention to gain an Adding value to overall view of the performance of NHS organisations. The interest of the performance Department of Health in exploring methods to aggregate performance measurement measures across the different performance areas of the performance assessment framework and in exploring the potential trade-offs between these areas has been present since its publication. In reality, some attempts have 1259 already been made to design aggregate performance measures in the health care sector. The composite performance indicator for health authorities and boards in England, Wales and Scotland developed by the King’s Fund is an example (Appleby and Mulligan, 2000). There are numerous approaches to aggregating multidimensional performance measures into a single indicator of performance for the organisational system as a whole. Each has different advantages and disadvantages. MCDA approaches have proved their potential in informing decision making through the integration of multiple heterogeneous measures into a single or a few key indicators of overall performance and in addressing trade-offs among measures, therefore, their use is well suited in the performance measurement context. The procedure we propose to assess how well the organisation is performing against each individual measure and to arrive at an indicator of overall performance makes use of a hierarchical, weighted additive value function and is supported by the use of the multicriteria decision support system visual interactive sensitivity analysis (VISA). The multi-attribute value function approach is well-grounded in theory and has long shown potential in the field of performance measurement (see, for example, Felix and Riggs, 1983). The reader is referred to Keeney and Raiffa (1976) and Von Winterfeldt and Edwards (1986) for a detailed description of the multi-attribute value function approach and to Belton and Vickers (1990) for an overview of the use of a simple multi-attribute value function incorporating VISA. This approach calls for the performance measures to be structured as a hierarchy, or performance measures tree, as depicted in Figure 4a for the illustrative example. At the first level of the tree is the composite indicator of overall performance of the hospital trust which is a function of the four fundamental areas or dimensions of performance. At the second level there are the four areas of performance: clinical effectiveness and outcomes, efficiency, patient/carer experience, and capacity and capability. Finally, at the third level there are the performance measures that contribute to each of these areas. The tree should emerge from the structuring process described in section 3.3.1, and is similar in form to the performance measurement structure adopted in other approaches (see for example Kaplan and Norton, 1992). However, it is recognised that moving from a causal loop structure to a tree structure is not necessarily straightforward (Belton et al., 1997) and this is an area of ongoing research. Principles of good practice in the structuring of value trees, which apply equally in the performance measurement context, are outlined by Keeney and
IJOPM 22,11
1260
Figure 4. VISA analysis
Raiffa (1976). These highlight the necessity of compromise between completeness and usability, the need for value independence and avoidance of double-counting. In practice it is likely that several iterations will be required to arrive at a tree which is usable and acceptable to all stakeholders. A further requirement is that the bottom-level criteria should represent concepts which are measurable and understandable to all potential users. If this requirement is met it should then be a relatively straightforward task to determine how each organisation performs against each individual measure, the process of scoring. There are many possible approaches to scoring, but all call for the clear definition of two reference points on the utilised scale (a natural scale may be used, in which case the reference points are the natural zero and unit of measurement). A common and easily used procedure is to compare the actual performance of a hospital trust against the targets defined (based on a range of acceptable performance) and score it using a normalised 0-100 global scale on which the 0 and 100 points are defined by the lowest tolerable and best
attainable possibility for each performance measure. The scoring process, Adding value to illustrated in Figure 4b, transforms a scaled level of performance on the natural performance scale for each measure to a scaled level of performance on the common scale, measurement and can be realised through direct rating or by using value functions. A value function or preference function defines the nature of the decision maker’s preference for a particular measure, which may be non-linear. There are a 1261 variety of procedures that can be utilised to develop these preference functions (for an overview, see Belton and Stewart, 2002). Independently of the procedure selected, it is important that the evaluation of the organisation with respect to the performance measures identified and diagrammed in Figure 4a is carried out by those with expert knowledge in the area wherever appropriate. In a decision-making context, often the process of scoring will call for judgement, as the ‘‘measures’’ in use have an element of subjectivity, and discussion of this further enhances participants’ understanding of each others’ perspectives. Having scored the performance of the hospital trust with respect to all the measures at level 3 of the performance measures’ tree, the next stage is to weight those measures to reflect their relative importance to the corresponding performance dimensions at level 2 (Figure 4c). These weights reflect acceptable trade-offs among measures and are elicited from decision makers in accordance with their preferences and priorities for each performance measure and dimension based on well-established axioms of decision making. The weights represent subjective judgements and thus are likely to be imprecise and potentially volatile. It is very likely that different stakeholder groups will want to weight performance measures differently. However, an important feature of the software used to support the process (VISA) is, as the name suggests, that it allows interactive exploration of changes in inputs to the model, with the impacts being seen immediately in easy to interpret visual format. This sensitivity analysis can be used to explore the robustness of ‘‘conclusions’’ to priorities and values of different stakeholders, and when ‘‘disagreement’’ exists to highlight the rationale for this, thereby furthering discussion and understanding of others’ perspectives. The ability to ‘‘play’’ with the model is also a powerful means of building confidence in it and understanding of it. Regarding scoring from a technical perspective, there are many possible ways of eliciting the weights for performance measures. Independently of the procedure to be adopted, it is important that people take into account the range over which the measure is assessed when assigning importance weights (failure to do so is common in many simple scoring and weighting methods and can easily be shown to lead to anomalous results). The weights for the higherlevel measures in the value tree, that is for the performance dimensions, can be found either by direct comparison of the performance dimensions at level 2 or by selective comparisons of performance measures at level 3. Once these weights are defined (Figure 4d), we are in a position to see how well the hospital trust performs in each performance dimension and get an indication of overall performance. This is done by using a hierarchical
IJOPM 22,11
weighted value function, of which the simplest, and most widely used form is the additive model: m X V ðaÞ ¼ wi vi ðaÞ i¼1
1262
where V(a) is the overall value or performance of the organisation being assessed; vi(a) is the value score reflecting organisation’s performance on measure i; and wi is the weight assigned to reflect the importance of measure i. The profile graphs in Figure 4c (upper diagrams) are an important tool to visualise how a hospital performs against the targets and in comparison with other units. Each performance measure (or dimension) is represented by a vertical bar and the performance profile of a hospital trust is shown by a coloured line crossing these bars at the appropriate point. Thus, it is easily seen if a trust has a good (or poor) all-round performance, or combines significant strengths in some areas with weaknesses elsewhere. When the scores of each performance dimension are combined with the respective weights, we obtain the overall performance score of the organisation (Figure 4d). However, as already cautioned, it is important not simply to take this number as ‘‘the answer’’, but to accept it as part of the overall learning process, another step in furthering understanding and promoting discussion about the problem. The process of measuring organisational performance in this way offers several advantages. First, it is founded on assessment of how well an organisation is performing against each individual measure, and as we can see in Figure 4b, scores can be directly related to natural measures and when appropriate reflect non-linearities of value which may exist. Second, the multiattribute value function approach is a transparent and easy-to-use technique that allows the integration of heterogeneous measures into a single or a few key indicators of overall performance. The decision makers can easily see how, via the aggregative model, their beliefs and preferences turn into composite indicators of overall performance. Third, trade-offs between measures are explicitly addressed. Given that the performance measures represent the interests of different stakeholders, the scoring and weighing of these measures is normally a collective process that provides an excellent opportunity for the working group to learn about the problem situation and about the values and priorities of the different participants. Finally, the use of a MCDA approach enables the decision makers to develop an explicit evaluation process, which might be used to justify and explain to others why a particular level of performance is observed. This procedure of measuring performance, however, is not without its own limitations and weaknesses. First, it is inherently subjective, in the choice of measures, in the weighting of measures and potentially also in the scoring process (if non-linear value functions are specified). Many critics cite this as a limitation of the approach. However, proponents would argue that all such decision making is subjective and that the value of MCDA is in making that
subjectivity explicit and integrating it in a transparent way with objective, Adding value to measurable, data. Second, there is no doubt that the procedure proposed is performance founded on some strong and restrictive assumptions about the decision measurement maker’s preferences, including transitivity, preference independence and summation. However, these assumptions are made to guide the decisionmakers to a more consistent and rational thought process. Third, the amount of 1263 time and effort required to establish and implement a sound system should not be underestimated and it is advised that the process is only appropriate for ‘‘decisions that matter’’ (Belton and Stewart, 2002). Once again, the engagement of a skilled facilitator/analyst who is knowledgeable about the methodology and skilled in managing group processes would be strongly advised. Overall, however, while limitations and implementation challenges exist the use of the MCDA procedure to assess performance is very valuable and should be encouraged as a forum for discussion and consequently, as a means of improving understanding about the problem faced, about the values and objectives of the different stakeholders and about organisational priorities. 3.3.3 Analysis. The measurement procedure using VISA described above, allows us to look individually at the scores on each of the performance measures and to synthesise that information in a way which indicates how the organisation performs on each of the key performance dimensions and overall. Whilst this information is very valuable for monitoring the performance of an organisation and indicating where improvement is required, it is of little or no help in driving hospital trusts if it is seen as an end in itself. To be effective, a performance measurement and management system has to support decision making, informing decision makers, among other things, about the reasons for poor (or good) performance and which actions to implement to obtain effective and appropriate change. Many performance measurement systems have fallen short of their potential because of failure to provide decision makers with the understanding and support necessary to do this. Unless this understanding is developed significant behavioural change will not occur and because symptomatic interventions are likely to be undertaken the measurement process will have limited impact. However, identifying the causes of problems and developing appropriate solutions is frequently a difficult process for the unaided decision maker. The system’s perspective tells us that we must look beyond symptoms to understand important problems and to find fundamental solutions (Forrester, 1961; Senge, 1990). Only a thorough analysis of the underlying structure of problems allows us to find their real causes and the areas of highest leverage, that is, those areas where actions can lead to lasting and significant performance improvements. To get insights into the real causes of problems and to find the highest leverage areas, Senge (1990) suggests that people must learn to see ‘‘structures’’ rather than ‘‘events’’ and think in terms of ‘‘processes’’ rather than ‘‘static snapshots’’. SD has repeatedly been demonstrated to be an effective analytical tool in fostering this way of thinking. The focus on the causal structure of problems and the search for leverage points are some of the
IJOPM 22,11
1264
strengths which make SD an appropriate approach to foster understanding of the process underlying performance generation and to identify the factors that may prompt effective changes in the system. Kaplan and Norton (2001) have themselves recognised the value of system dynamics modelling to increase understanding of a business’ value creation process. The use of qualitative SD based on causal loops (see Figure 3) and quantitative SD based on computer simulation (see next section) can be, therefore, very valuable in assisting decision makers to gain a greater understanding about how the organisation is performing, why, and what to do to improve results. In section 3.3.1 we discussed some of the insights that can be gained from developing CLDs; however, as stressed there, caution needs to be taken in inferring behaviour from CLDs alone. In most cases, quantitative simulation models are essential to understand fully the dynamics of complex systems. In addition, the use of SD simulation models not only provides understanding of what has happened but also can generate valuable insights about what might be about to happen next. 3.3.4 Improvement. SD modelling has been used and proved to be very valuable in several areas of health care policy and health care management, particularly when used as a learning tool (see, for example, Dangerfield and Roberts, 1999; Lane et al., 2000). The support to group learning that SD modelling can provide (Vennix, 1996) can also be invaluable in the performance measurement and management context. The model building process clarifies thinking and encourages the different system stakeholders to learn about the system being assessed as a whole. In addition, SD simulation modelling may play a vital role in testing and comparing alternative actions to improve system’s performance. In some cases, if participants do not have access to a simulation model to test and design policy actions there is the danger that the selected policies will worsen the problem instead of amending it. In other cases, even if a given situation improves, as a result of adopting a given course of action, it can be only temporary, and consequently, other policies could have been more effective. To understand the dynamic complexity inherent in these situations, a SD simulation model is required. By developing and running this model participants can understand the stock and flow structure of systems and observe the changes that occur over time in the variables of interest. Moreover, in developing a SD simulation model the system’s internal structure is explicitly represented and this structure is often the underlying source of the problem. Thus, by finding and modifying the system structure decision makers are able to improve organisational performance in the most effective way. However, the selection of the ‘‘best’’ action plan is not straightforward. Stakeholders have different and often conflicting objectives and, as a result, trade-offs must be made. Given that the decision maker is confronted with a large and complex amount of information and given that it is likely that none of the alternative courses of action optimises all performance measures, the use of an appropriate MCDA approach can also be very valuable to assist the decision process and to enhance understanding about the problem situation.
As observed in section 3.3.2, the results of measurement indicate that the Adding value to example hospital trust is performing poorly with respect to the percentage of performance patients waiting less than 6 months for an inpatient admission. How can the measurement hospital reduce waiting lists and ultimately the time patients have to wait for admission? This is a complex (and highly topical) problem and one for which there is no simple solution. This issue clearly involves multiple, conflicting 1265 objectives and it is very likely that there will be alternative courses of action or strategies to be considered. For example, to reduce the size of inpatient waiting lists, a possible action is to reduce the length of stay in hospital. However, this would increase the likelihood of inappropriate discharges and, consequently, would increase the number of emergency re-admissions. That is, to improve one performance measure (for example, the percentage of patients waiting less than six months for admission) we have to sacrifice the performance of another measure (for example, the emergency re-admission rates). That is to say that the improvement in the efficiency of the hospital trust would be at the expense of the clinical effectiveness and outcomes of treatment. Another possible action is to increase bed capacity in the hospital. Often, however, the proposed action can have consequences that are far from the intended effect and therefore, to understand fully the effects of these actions a SD simulation model may be required. For example, Wolstenholme (1999) demonstrates through the use of SD modelling that an increase in hospital bed capacity is not the most effective solution to reduce total patient waiting times. He illustrates that, when bed capacity is increased, although more patients are admitted to hospital, the effect is temporary. As soon as the new capacity is full, the number of patients in hospital wards stabilises and the pre-hospital waiting time increases again. Furthermore, a counter-intuitive behaviour may occur as a consequence of this type of policy. For example, the additional bed capacity introduced can stimulate more demand for hospital treatment or, at least, encourage more GP referrals to hospital. Figure 5a shows two hypothetical policy alternatives and some of the impacts to be evaluated. As we can see, although the model indicates that both alternatives lead to an increase of the percentage of patients waiting less than 6 months for admission compared with the current (actual) position, they simultaneously increase the number of emergency re-admissions. It is often the case that none of the alternative courses of action considered will generate improvement on all performance measures, given that some of them are conflicting. In order to evaluate and decide which policy alternative is better, the MCDA procedure previously discussed can be applied. The process of evaluation calls for the acquisition of two types of information. First, information reflecting the performance of the alternatives under consideration in the light of the identified criteria. In the present context, this information results from running the simulation model and observing how the different plans of action proposed by those with expert knowledge in the area perform in each of the measures considered (Figure 5a). Second, there is
IJOPM 22,11
1266
Figure 5. Portion of a flight simulator and of the VISA analysis
information capturing feelings about the relative importance of each performance measure. Once this information is collected, it is synthesized using the multi-attribute value function supported by the software VISA. As can be seen in Figure 5b this procedure allows us to compare the actual performance (base case) of the hospital and the performance of the two policy options under consideration against the targets set by decision makers and to know which of the proposed action plans is preferred. Figure 5b shows how each alternative performs against each measure, dimension and overall. As the results show, each of the options under consideration presents strengths and weaknesses. It is not uncommon when using a MCDA approach that the analysis prompts the decision makers to identify and implement alternatives not previously considered. For example, it may happen that a new option resulting from the combination of the existing ones will be preferred. That is, cycling between the SD model and the MCDA procedure, decision makers can enhance their understanding about the problem and end up with a better outcome. The use of the SD and MCDA approaches in this stage offers two obvious advantages. From one side, SD modelling allows decision makers to develop a
holistic understanding of the problem and to verify, through the use of Adding value to ‘‘microworlds’’, and consequently, in a risk-free environment, the effect of performance different actions on a system’s performance measures over time. From another measurement side, using MCDA, decision makers can develop an explicit evaluation process of those actions. In summary, the use of SD and MCDA can make the performance 1267 measurement and management process more efficient and effective by increasing the information processing capabilities of the decision makers and by enabling them to do a more thorough problem analysis than would be possible without their use. 4. Closing remarks It has long been recognised that performance measurement is vital to the success and development of most organisations. Managers in both the private and public sectors are under constant pressure to improve organisational performance. However, improving the performance of an organisation is not a simple matter. Instead, it is frequently a complex and poorly defined problem, the solution of which often requires a process of organisational learning enabling decision makers to change the way they think and act, and consequently, enabling a more effective use of the available information. While researchers and practitioners in the field of performance measurement have made considerable progress in recent years towards a more effective development and use of measurement systems, there still remain some issues which prevent organisations getting the most from their performance measurement initiatives. Particularly, it is recognised that approaches which allow decision makers to increase their understanding about the process underlying performance generation, to understand the implications of alternative courses of action before they become operational and which help them in evaluating and eventually selecting appropriate corrective actions, can provide very valuable insights when supporting the process of performance measurement and management. SD and MCDA are two approaches to modelling that have individually proved their potential to inform and support decision making. This paper has sought to demonstrate that there is also a clear potential for these approaches to be employed in support of performance measurement and management. While we do not claim the use of SD and MCDA is the solution to all the issues discussed in this paper, it can bring new insights to inform and support the different stages of the performance measurement and management process. As far as the design of measurement systems is concerned, the use of SD and MCDA modelling provides a focus for discussion and a shared language between the different individuals and groups of decision makers, which facilitates communication and learning. Throughout the modelling process, the participants have to make explicit and share the mental models they have as well as the priorities and values that influence their decision making. This will provide a powerful means for exploring and reconciling important differences
IJOPM 22,11
1268
in the models and value systems held by the different participants, increasing consensus, ownership and commitment among them and leading to a better understanding of the situation and of each other’s perceptions of it. This understanding is very important to decide what to measure and why and may prove invaluable for the successful implementation of the measurement system. With respect to the measurement of performance, the use of a MCDA approach requires the participants to make explicit the relative importance of the different performance measures through the specification of acceptable trade-offs between these measures. This fosters understanding about the problem, about the values and objectives of the different stakeholders and about organisational priorities. By using a MCDA procedure the members of the working group can easily see (and justify to others) how their beliefs and preferences turn into a single or few indicators of overall performance. In addition, a thorough sensitivity analysis can be carried out to analyse how robust these indicators are to changes on the priorities and values of the different stakeholders. With regard to the analysis and use of performance results, and as a consequence of the previous points, SD and MCDA modelling offer a powerful frame in analysing why a particular level of performance is observed and the ways by which changes in system’s performance occur. Furthermore, because SD modelling focuses on the causal structure of problems and because MCDA enables decision makers to take explicitly into account multiple and conflicting criteria in the decision-making process, they assist decision makers not only to understand better why the organisation is performing in a certain way but also to develop and eventually select better informed and justifiable improvement solutions. That is, the integrated use of SD and MCDA makes the policy making process for improving the performance of the organisation more efficient and effective by enabling decision makers to analyse and evaluate rapidly, but thoroughly, in a risk-free environment the effects of different actions on the system’s performance. Finally, and as previously discussed, it is important to note that the use of these approaches to support the process of performance measurement and management is not without its own weaknesses. Overall, however, while limitations and implementation challenges exist, the authors’ prior experiences in using these approaches independently, taken together with external responses to the initial research proposal presented here, indicate that these can be addressed and that, in appropriate circumstances, significant benefits can be gained. As indicated in the introduction, the next phase is one of action research, a test of the proposed methodology in an organisational context. This research will seek to address a number of issues, including: . The value of the use of SD and MCDA independently and in an integrated manner in enhancing understanding of issues and in supporting performance measurement and management.
.
.
The nature of the practical difficulties and challenges that arise in Adding value to applying SD and MCDA (independently and in integration) – both performance anticipated and unexpected. measurement The effectiveness of proposed theoretical and technical links between the methods and supporting softwares (although not a focus of this paper this has been part of the research to date).
1269
The research will employ both qualitative and quantitative methods of data collection including observation of workshops, interviews with involved parties and tracking of model development. The information will be used to identify critical learning incidents and points of difficulty as well as holistic evaluations. The longer term aim is to refine the methodology, including exploration of its integration within established frameworks for performance measurement such as the balanced scorecard (discussions with a colleague who has substantial practical experience in its use are ongoing), and to compare and contrast the benefits and problems that arise from its application in different organisational contexts. References Appleby, J. and Mulligan, J. (2000), ‘‘Performance measures: the public’s choice’’, in Appleby, J. and Harrison, A. (Eds) , Health Care UK, King’s Fund, London, pp. 59-63. Ballantine, J., Brignall, S. and Modell, S. (1998), ‘‘Performance measurement and management in public health services: a comparison of UK and Swedish practice’’, Management Accounting Research, Vol. 9, pp. 71-94. Banks, R.L. and Wheelwright, S.C. (1979), ‘‘Operations vs strategy: trading tomorrow for today’’, Harvard Business Review, May-June, pp. 112-20. Belton, V. (1985), ‘‘The use of a simple multiple-criteria model to assist in selection from a shortlist’’, Journal of the Operational Research Society, Vol. 36, pp. 265-74. Belton, V. (1990), ‘‘Multiple criteria decision analysis: practically the only way to choose’’, in Hendry, L. and Eglese, R. (Eds), Operational Research Tutorial Papers, Operational Research Society, Birmingham, pp. 53-101. Belton, V. (1993), ‘‘Project-planning and prioritization in the social services – an OR contribution’’, Journal of the Operational Research Society, Vol. 44 No. 2, pp. 115-24. Belton, V. and Elder, M. (1994), ‘‘Decision support systems: learning from visual interactive modelling’’, Decision Support Systems, Vol. 12, pp. 355-64. Belton, V. and Stewart, T.J. (2002), Multiple Criteria Decision Analysis: An Integrated Approach, Kluwer Academic Publishers, Boston, MA. Belton, V. and Vickers, S. (1990), ‘‘Use of a simple multi-attribute value function incorporating visual interactive sensitivity analysis for multiple criteria decision making’’, in Bana e Costa, C.A. (Ed.), Readings in Multiple Criteria Decision Aid, Springer-Verlag, Berlin, pp. 319-34. Belton, V., Ackermann, F. and Shepherd, I. (1997), ‘‘Integrated support from problem structuring through to alternative evaluation using COPE and VISA’’ (Wiley Prize winning paper), Journal of Multiple Criteria Analysis, Vol. 6, pp. 115-30. Bititci, U.S., Turner, T. and Begemann, C. (2000), ‘‘Dynamics of performance measurement systems’’, International Journal of Operations & Production Management, Vol. 20 No. 6, pp. 692-704.
IJOPM 22,11
1270
Bourne, M., Mills, J., Wilcox, M., Neely, A. and Platts, K. (2000), ‘‘Designing, implementing and updating performance measurement systems’’, International Journal of Operations & Production Management, Vol. 20 No. 7, pp. 754-71. Coughlan, P. and Coghlan, D. (2002), ‘‘Action research for operations management’’, International Journal of Operations & Production Management, Vol. 22 No. 2, pp. 220-40. Coyle, R.G. (1984), ‘‘A systems approach to the management of a hospital for short-term patients’’, Socio-Economic Planning Sciences, Vol. 18 No. 4, pp. 219-26. Coyle, R.G. (1985), ‘‘The use of optimization methods for policy design in a system dynamics model’’, System Dynamics Review, Vol. 1 No. 1, pp. 81-91. Coyle, R.G. (1996), System Dynamics Modelling – A Practical Approach, Chapman & Hall, London. Da Silveira, G. and Slack, N. (2001), ‘‘Exploring the trade-off concept’’, International Journal of Operations & Production Management, Vol. 21 No. 7, pp. 949-64. Dangerfield, B. and Roberts, C. (Eds) (1999), ‘‘Health and health care dynamics’’, Special Issue of the System Dynamics Review, Vol. 15 No. 3. Department of Health (1997), The New NHS: Modern. Dependable, DoH, London, available at: www.open.gov.uk/doh/newnhs/newnhs.htm Department of Health (2000), The NHS Plan: A Plan for Investment; A Plan for Reform, DoH, London, available at: www.nhs.uk/nationalplan/ Department of Health (2001), NHS Performance Ratings: Acute Trusts 2000/01, DoH, London, available at: www.doh.gov.uk/performanceratings/ Department of Health (2002), NHS Performance Indicators: Acute NHS Hospital Trusts, DoH, London, available at: www.doh.gov.uk/nhsperformanceindicators/hlpi2002/ Dumond, E.J. (1994), ‘‘Making best use of performance measures and information’’, International Journal of Operations & Production Management, Vol. 14 No. 9, pp. 16-31. Eccles, R.G. and Pyburn, P.J. (1992), ‘‘Creating a comprehensive system to measure performance’’, Management Accounting, October, pp. 41-4. Eden, C. (1990), ‘‘The unfolding nature of group decision support – two dimensions of skill’’, in Eden and Radford (Eds), Tackling Strategic Problems: The Role of Group Decision Support, Sage, pp. 48-52. Eden, C. and Huxham, C. (1996), ‘‘Action research for management research’’, British Journal of Management, Vol. 7, pp. 75-86. Felix, G.H. and Riggs, J.L. (1983), ‘‘Productivity measurement by objectives’’, National Productivity Review, Vol. 2 No. 4, pp. 386-93. Fine, C.H. and Hax, A.C. (1985), ‘‘Manufacturing strategy: a methodology and an illustration’’, INTERFACES, Vol. 15 No. 6, pp. 18-46. Fitzgerald, L., Johnston, R., Brignall, S., Silvestro, R. and Voss, C. (1991), Performance Measurement in Service Businesses, CIMA Publishing, London. Flapper, S.D., Fortuin, L. and Stoop, P.P. (1996), ‘‘Towards consistent performance management systems’’, International Journal of Operations & Production Management, Vol. 16 No. 7, pp. 27-37. Forrester, J.W. (1961), Industrial Dynamics, MIT Press, Cambridge, MA. French, S. (1988), Decision Theory: An Introduction to the Mathematics of Rationality, Ellis Horwood, Chichester. Fry, T.D. and Cox, J.F. (1989), ‘‘Manufacturing performance: local versus global measures’’, Production and Inventory Management Journal, Second Quarter, pp. 52-7.
Gardiner, P.C. and Ford, A. (1980), ‘‘Which policy run is best, and who says so?’’, in Legasto, A.A., Forrester, J.W. and Lyneis, J.M. (Eds), System Dynamics: TIMS Studies in the Management Sciences, North-Holland, Amsterdam, pp. 241-57. Globerson, S. (1985), ‘‘Issues in developing a performance criteria system for an organization’’, International Journal of Production Research, Vol. 23 No. 4, pp. 639-46. Goodwin, P. and Wright, G. (1998), Decision Analysis for Management Judgement, Ch. 2, John Wiley & Sons, Chichester. Harrington, H.J. (1991), ‘‘Improving business processes’’, TQM Magazine, Vol. 3 No. 1, pp. 39-44. Howick, S. and Eden, C. (2001), ‘‘The impact of disruption and delay when compressing large projects: going for incentives?’’, Journal of the Operational Research Society, Vol. 52 No. 1, pp. 26-34. Ittner, C.D. and Larcker, D.F. (1998), ‘‘Innovations in performance measurement: trends and research implications’’, Journal of Management Accounting Research, Vol. 10, pp. 205-38. Kaplan, R.S. and Norton, D.P. (1992), ‘‘The balanced scorecard – measures that drive performance’’, Harvard Business Review, January-February, pp. 71-9. Kaplan, R.S. and Norton, D.P. (2001), The Strategy-Focused Organization – How Balanced Scorecard Companies Thrieve in the New Business Environment, Harvard Business School Press, Boston, MA. Keeney, R.L. and Raiffa, H. (1976), Decisions with Multiple Objectives: Preferences and Value Tradeoffs, Cambridge University Press, Cambridge. Kolb, D.A. (1984), Experiential Learning, Prentice-Hall, Englewood Cliffs, NJ. Lane, D.C., Monefeldt, C. and Rosenhead, J.V. (2000), ‘‘Looking in the wrong place for health care improvements: a system dynamics study of an accident and emergency department’’, Journal of the Operational Research Society, Vol. 51 No. 5, pp. 518-31. Lebas, M.J. (1995), ‘‘Performance measurement and performance management’’, International Journal of Production Economics, Vol. 41, pp. 23-35. Legasto, A.A. Jr and Maciariello, J. (1980), ‘‘System dynamics: a critical review’’, in Legasto, A.A. Jr, Forrester, J.W. and Lynes, J.M. (Eds), System Dynamics, TIMS Studies in the Management Sciences 14, North-Holland Publishing, Amsterdam, pp. 23-43. Lynch, R.L. and Cross, K.F. (1991), Measure Up! How to Measure Corporate Performance, Blackwell Publishers, Oxford. Mapes, J., New, C. and Szwejczewski, M. (1997), ‘‘Performance trade-offs in manufacturing plants’’, International Journal of Operations & Production Management, Vol. 17 No. 10, pp. 1020-33. Meadows, D.H. (1980), ‘‘The unavoidable a priori’’, in Randers, J. (Ed.), Elements of the System Dynamics Method, Productivity Press, Cambridge, MA, pp. 23-57. Meredith, J., Raturi, A., Amoako-Gyampah, K. and Kaplan, B. (1989), ‘‘Alternative research paradigms in operations’’, Journal of Operations Management, Vol. 8 No. 4, pp. 297-326. Morecroft, J. and Sterman, J. (Eds) (1994), Modeling for Learning Organisations, Productivity Press, Portland, OR. Neely, A. (1999), ‘‘The performance measurement revolution: why now and what next?’’, International Journal of Operations & Production Management, Vol. 19 No. 2, pp. 205-28. Neely, A., Adams, C. and Kennerley, M. (2002), The Performance Prism: The Scorecard for Measuring and Managing Business Success, FT Prentice-Hall, London. Neely, A., Gregory, M. and Platts, K. (1995), ‘‘Performance measurement system design – a literature review and research agenda’’, International Journal of Operations & Production Management, Vol. 15 No. 4, pp. 80-116.
Adding value to performance measurement 1271
IJOPM 22,11
1272
Neely, A., Mills, J., Platts, K., Richards, H., Gregory, M., Bourne, M. and Kennerly, M. (2000), ‘‘Performance measurement system design: developing and testing a process-based approach’’, International Journal of Operations & Production Management, Vol. 20 No. 10, pp. 1119-45. Neely, A., Richards, H., Mills, J., Platts, K. and Bourne, M. (1997), ‘‘Designing performance measures: a structured approach’’, International Journal of Operations & Production Management, Vol. 17 No. 11, pp. 1131-52. NHS Executive (1999), ‘‘Quality and performance in the NHS: high level performance indicators’’, NHS Executive, London, available at: www.doh.gov.uk/indicat/nhslpi.htm Pritchard, R.D., Roth, P.L., Jones, S.D. and Roth, P.G. (1990/91), ‘‘Implementing feedback systems to enhance productivity: a practical guide’’, National Productivity Review, Winter, pp. 57-67. Richardson, G.P. and Andersen, D.F. (1995), ‘‘Teamwork in group model building’’, System Dynamics Review, Vol. 11 No. 2, pp. 113-37. Richardson, G.P. and Pugh, A.L. III (1981), Introduction to System Dynamics Modeling with DYNAMO, Productivity Press, Cambridge, MA. Richardson, P.R., Taylor, A.J. and Gordon, J.R. (1985), ‘‘A strategic approach to evaluating manufacturing performance’’, INTERFACES, Vol. 15 No. 6, pp. 15-27. Senge, P.M. (1990), The Fifth Discipline: The Art and Practice of the Learning Organization, Doubleday Currency, New York, NY. Skinner, W. (1974), ‘‘The focused factory’’, Harvard Business Review, May-June, pp. 113-21. Sloper, P., Linard, K.T. and Paterson, D. (1999), ‘‘Towards a dynamic feedback framework for public sector performance management’’, International System Dynamics & ANZSYS Conference. Sterman, J.D. (1989a), ‘‘Misperceptions of feedback in dynamic decision making’’, Organizational Behavior and Human Decision Processes, Vol. 43, pp. 301-35. Sterman, J.D. (1989b), ‘‘Modeling managerial behavior: misperceptions of feedback in a dynamic decision making experiment’’, Management Science, Vol. 35 No. 3, pp. 321-39. Sterman, J.D. (2000), Business Dynamics Systems Thinking and Modeling for a Complex World, McGraw-Hill, London. Suwignjo, P., Bititci, U.S. and Carrie, A.S. (2000), ‘‘Quantitative models for performance measurement system’’, International Journal of Production Economics, Vol. 64, pp. 231-41. Vennix, J.A.M. (1996), Group Model Building: Facilitating Team Learning Using System Dynamics, John Wiley & Sons, Chichester. Von Winterfeldt, D. and Edwards, W. (1986), Decision Analysis and Behavioral Research, Cambridge University Press, Cambridge. Wolstenholme, E.F. (1999), ‘‘A patient flow perspective of UK Health Services: exploring the case for new ‘intermediate care’ initiatives’’, System Dynamics Review, Vol. 15 No. 3, pp. 253-71.
The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/0144-3577.htm
Web enabled performance measurement systems
Web enabled measurement systems
Management implications U.S. Bititci, S.S. Nudurupati and T.J. Turner
1273
Centre for Strategic Manufacturing, University of Strathclyde, Glasgow, UK, and
S. Creighton Alcan Foil Europe, Glasgow, UK Keywords Performance measurement, Business culture, Information technology, Flexibility, Internet Abstract In order to respond proactively to the new emerging challenges, management requires up-to-date and accurate information on performance. Such performance measurement systems are still not common because there is not sufficient research focused on management implications of IT enabled performance measurement systems. This paper presents an empirical case study where a fully integrated IT enabled performance measurement system was implemented and its management implications studied. The paper concludes, based on a single case study, that if properly implemented, such systems would promote a proactive management style and greater confidence in management decisions.
Introduction The manufacturing industries are facing tough challenges to succeed in a globally competitive market. Markets are becoming increasingly sophisticated and dynamic. Customer demand is changing rapidly in terms of sophistication of the products and services they require. As a result, companies need to become more responsive to customer and market needs with greater numbers of customer-specific products, more flexible processes, suppliers and resources co-ordinated through a number of factories and warehouses, while reducing costs. In order to respond proactively to these challenges, management requires up-to-date and accurate information on its: . markets and customers; . competitive position; . financial performance; . customer service performance; . operational performance; . suppliers’ performance, and so on. Furthermore, this performance information needs to be integrated, dynamic, accurate, accessible and visible to aid fast decision making to promote a proactive management style leading to agility and responsiveness.
International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, pp. 1273-1287. # MCB UP Limited, 0144-3577 DOI 10.1108/01443570210450310
IJOPM 22,11
1274
Recent years have seen the development of a number of frameworks and models for performance measurement. Some companies are now attempting to make use of information technology to provide the required information online. However, despite this amount of research and development, performance measurement systems that are properly integrated, dynamic, accurate, accessible and visible to facilitate responsive manufacturing, are still not common (Bititci and Carrie, 1998). This is because the technical, technological and people issues concerning dynamics of performance measurement systems are not completely understood. The objective of the research presented in this paper was to study the management issues concerning dynamics of performance measurement by embedding performance measurement systems using IT and Web technology, implementing these in collaborating companies and studying its impact on performance, people and decision making. Background The background of this paper extends back to the mid-1980s when the need for better integrated performance measurement systems was identified (Johnson and Kaplan, 1987; McNair and Masconi, 1987; Kaplan, 1990; Druker, 1990; Russell, 1992). Since then, there have been numerous publications emphasising the need for more relevant, integrated, balanced, strategic and improvementoriented performance measurement systems. In terms of frameworks and models, the SMART model (Cross and Lynch, 1988-1989) and the performance measurement questionnaire (Dixon et al., 1990) were developed in the late 1980s. In the 1990s the balanced scorecard (Kaplan and Norton, 1996) made a significant impact by creating a simple, but effective, framework for performance measurement. During the 1990s the European Business Excellence Model (EFQM, 1998) also made a significant impact on what measures companies used and what they did with these measures. The EPSRC-funded research on integrated performance measurement systems tested the feasibility of developing an auditable reference model from three different viewpoints – structures (Bititci and Carrie, 1998), information (Kehoe and Little, 1998) and people behaviour (Burns and Backhouse, 1998). This work built on the balanced scorecard and EFQM models using the viable systems structure (Beer, 1985) and resulted in the development of the integrated performance measurement systems reference model. Other research programmes, and to a certain extent consultancy organisations, also developed approaches, procedures and guidelines for developing and designing effective performance measurement systems (Doumeingts et al., 1995; Krause, 1999). The Performance Measurement Workbook developed at Cambridge by Neely et al. (1996) is now widely known, and the performance PRISM (Neely and Adams, 2001) is emerging as the most contemporary framework for performance measurement. There have been several other initiatives for developing and defining performance measures for various business areas and processes, including
performance measures for production planning and control (Kochhar et al., 1996), performance measures for the product development process (Oliver, 1996) performance measures for planning and control (Oliver Wight, 1993). Bititci and Carrie (1998) used the IPMS reference model to audit the performance measurement systems of over 30 UK and European-based manufacturing and service companies. This work identified the main reasons behind the absence of performance measurement systems that would facilitate responsiveness and agility as: . Today most performance measurement systems are historical and static. That is, they are not dynamic and sensitive to changes in the internal and external environment of the firm. As a result, the information presented is not relevant, up-to-date or accurate. This creates a vicious circle because it has a negative effect on the perceived value and usefulness of the performance measurement system, resulting in lack of commitment and ownership, which, in turn, discourages proper maintenance and updating of the system. . Few performance measurement systems have an integrated IT infrastructure. This results in cumbersome and time-consuming data collection, sorting, maintenance and reporting. As a result, companies cannot justify further investment of already stretched resources in data collection, sorting, maintenance and reporting type activities, which have low perceived values. Bourne and Neely (2000) independently monitored reasons behind the success and failure of performance measurement systems implementations and concluded that the design, implementation and maintenance of performance measurement systems are seen to be time-consuming activities, and that the benefits of these activities could not be clearly justified. Similarly, Hudson et al. (1999) summarised the reasons for failure of performance measurement systems initiatives in SMEs as: . historical measures with out-of-date and irrelevant information; . the time consuming nature of PMS development and maintenance; . lack of commitment and enthusiasm from senior management. It seems that three independent research projects (Bititci, 2000; Bourne and Neely, 2000; Hudson et al., 1999) all identified one of the barriers affecting the successful implementation of performance measurement systems as the time consuming and cumbersome nature of data maintenance associated with manual or partially IT-supported performance measurement systems. These researchers also found that this manifested as performance measures with outof-date and irrelevant information leading to lack of commitment and enthusiasm at all levels. Some IT vendors and consultants have also developed software tools for designing, implementing and managing performance measurement systems,
Web enabled measurement systems 1275
IJOPM 22,11
1276
such as: IPM, pbViews, iThink Analyst and PerformancePlus (Coleman, 1998). In fact some ERP vendors, such as Oracle, are at present in advanced stages of developing performance measurement modules to integrate with and support their ERP modules. Further research done by Begemann and Bititci (1999) established that use of IT platforms make outputs from performance measurement more accessible and visible, as well as making updating and maintenance of performance measures easier. However, there is little empirical research that supports the proposition that a fully integrated IT-supported performance measurement system would promote a more proactive and agile management style by providing dynamic, accurate and readily accessible information to aid decision making. The research The objective of the research presented in this paper was to study the management implications of Web-enabled performance measurement systems. This was achieved by: . Working with Alcan Packaging, Foil Rolling and Technical Products (AFE), UK, to develop a fully integrated Web-enabled, intranet-based, performance measurement system based on the integrated performance measurement systems (IPMS) reference model (Bititci and Carrie, 1998). . Creating a structured data collection tool to assess the management implications before, during and after implementation of the Webenabled performance measurement system. Architecture of the Web-enabled performance measurement system (WePMS) Alcan Packaging, Foil Rolling and Technical Products (AFE), is a profit centre of Alcan. The site specialises in the manufacture of thin aluminium foil (such as those used for cooking, food and confectionery packaging) and laminated aluminium foil (such as those used in the cigarette packaging). The main processes within the factory are rolling and laminating. The IPMS reference model (Bititci and Carrie, 1998) was used to guide the management team to identify and structure the key performance measures. The details of the structure of the performance measurement system has been the subject of previous papers, thus it has not been included in this paper. At the heart of the WePMS implemented at AFE is the Quality Analyst software product, which is essentially a software platform designed specifically for supporting statistical process control applications. It provides a good combination of flexibility, and ease of use. It easily integrates into manufacturing data systems. It produces all standard SQC charts, plus many special-purpose charts. It allows the user to dig into the information by ‘‘drilling down’’ into the charts. With the Open Data Base Connectivity (ODBC), Quality Analyst can pull down data from different sources into its own database. In this case, it is being
used as a tool to collect and convert numerical data into graphical Shewhart charts (more commonly known as SPC charts). In AFE the numerical data is available from a number of sources including: . MRPII system; . spreadsheet applications, i.e. MS Excel; . database applications, i.e. MS Access; . machine controllers, i.e. the process controllers of various equipment, such as mills; . data loggers.
Web enabled measurement systems 1277
Figure 1 illustrates this structure where Quality Analyst provides the main interface between the Web pages and the numerical data. This architecture enables the following: . Creation of data either automatically (through the MRPII system, machine controllers and the data loggers) or manually (through spreadsheets and databases). . Annotation of data either manually (e.g. by providing a comment field within the database or spreadsheet) or automatically (through reason codes that may be available within the existing systems). . Viewing information through AFE’s intranet pages. In AFE the WePMS is known as the management team reporting (MTR) system. Figure 2 illustrates a sample of the performance reports available through the MTR system. The Web page in Figure 2a illustrates the main menu page providing access to performance reports for various parts of the business. In Figure 2b, the top chart illustrates a Shewhart chart for monitoring variations in the porosity during the rolling process. This is a critical technical measure captured by the data loggers on a continuous basis and the daily averages are reported automatically by the MTR system at the end of each day. The figure also illustrates that the tolerances were tightened in July 1999. Annotations communicate that the targets are not achieved, indicating the
Figure 1. Architecture of the Web-enabled performance measurement system at AFE
IJOPM 22,11
1278
Figure 2. Sample pages from the AFE’s MTR system (WePMS)
reasons such as paper change, roll change, etc. The bottom charts illustrate the process capability histograms, one before tightening the tolerances and the other after tightening the tolerances. The Web page in Figure 2c illustrates AFE’s performance with respect to customer complaints received each month. This information is captured manually. As customer complaints are received by the customer services department they are entered into a customer complaints log (MS Access Database), in accordance with the company’s ISO9000 procedures. Quality Analyst picks this information from the database and presents it on the Web page. The chart in Figure 2c shows customer complaints received by AFE. Figure 2d illustrates two charts, one illustrating backlog against customer orders and the other illustrating delivery performance against customer orders. The chart shows that delivery performance has been within target of 95 per cent except between weeks 36 and 41, where delivery performance has suffered considerably due to a problem with raw material supply (as annotated). Evaluating the management implications of WePMS Throughout the research, the researchers played a dual role. Crigeton was one of the managers within Alcan responsible for specifying, developing and implementing the fully integrated Web-enabled, intranet-based, performance measurement system. Turner was the researcher acting in an advisory capacity
assisting with the implementation of the IPMS-based performance measurement system. Bititci and Nudurupati were acting as external observers, who conducted the empirical research independently of the development and implementation process. At the outset of the research, a number of interviews were held with the senior management and middle management teams, which led the research team to understand the before scenario at AFE. During the design, development and implementation of the new performance measurement system, Creighton and Turner were observing and recording the implementation process, which was reviewed by the senior management team and the research team on a quarterly basis, through project steering committee meetings. The after scenario was captured as a result of a set of structured interviews with the key users of the system. This involved all levels of management from the general manager to team leaders. The results presented in this paper are thus based on a population of 29 managers, which remained stable throughout the research with no changes (i.e. nobody leaving or joining the company). In order to evaluate the business and management implications of the WePMS, a set of research questions were formulated at the outset, based on the background discussion provided earlier in the paper. These questions included: . What is the cost of implementing and operating WePMS? . What are the business benefits of implementing and operating WePMS? . What is the impact of the WePMS on business performance and strategy? . Are managers more confident in their decisions as a result of WePMS? . Are the managers and decisions becoming more proactive as a result of using WePMS? . How does the WePMS affect the management behaviour? . How does the WePMS affect the dissemination of knowledge throughout the organisation? . Does the WePMS impact on the visibility of information throughout the organisation? . Does the WePMS highlight or pinpoint the weaknesses of the business? The interviews for the before and after scenarios were structured to capture the management opinion with respect to the above questions through a simple but structured questionnaire. During the after interview each respondent was asked to rate the impact of the system, in the context of each question, on a fivepoint scale (significant improvement, some improvement, no improvement, some deterioration, significant deterioration). The respondents were also asked to justify their choice by explaining the key features of the change. The data was collected through face to face interviews with a cross-section of personnel in AFE.
Web enabled measurement systems 1279
IJOPM 22,11
1280
Research results In this section the feedback obtained from the questionnaire and personal interviews is summarised under separate headings corresponding to questions asked. Cost and scope of implementation The company has invested approximately £20,000 on buying and implementing the software, which includes the cost of the software products and training. It has invested £50,000 in hardware to support the software, which includes new PCs, data loggers, etc. The company has employed one full time employee and allocated, on average, approximately 10 per cent of the management time to this project. Most of the interviewees responded that it is very difficult to quantify the operating costs, as it has become a routine part of their business. Business benefits of implementing and operating the MTR system Even though few respondents indicate that there are some benefits, many respondents indicate that there are significant benefits. The MTR system has created visibility for promoting continuous improvement. It identified, strengths, weaknesses, opportunities and threats for the business: (1) Before: . Limited transparency of information at all levels. . Duplication of data (often inaccurate). . Measures with no clear targets and focus on what needs to be done. . Many people are involved in collecting data. (2) After: . There is complete transparency and visibility of information. . Data is now from a single source and accurate. . Problems are clearer, which helps to focus and manage improvement. . The cause and effect relationships between capital targets and operational activities are more visible, making monitoring against capital objectives easier and more visible. . Everyone knows what needs to be done and problems are not hidden. Impact on business performance and strategy Even though some people responded that there is some improvement, many people responded that there is significant improvement. The consensus view was that the business performance would have stagnated if the system was not in place. They said that: (1) Before: . Managers found it difficult to focus on key areas of business improvement.
.
‘‘Flavour of the month’’ approach to improvement.
.
Unstructured approach to performance planning.
(2) After: .
Now completely focused on areas needing improvement.
.
There is a systematic approach to improvement.
.
.
.
The capabilities of business and manufacturing processes are better understood. Achieved state of excellence on certain quality characteristics, which allowed adoption of higher technical specifications resulting in improved margins. Improved partnership with major customers who can view the MTR system during audits and see that the issues concerning business problems are being addressed.
In one particular case, a customer was ready to take its business elsewhere. Having seen how AFE was planning to tackle the problem, they got involved in the process. As a team they improved the performance of the product using the MTR system to such an extent that the customer was delighted with the results. They have now formalised a partnership agreement for AFE to supply other products. Confidence in management decisions Most of the respondents have reported that they are now more confident in their management decisions. They said that: (1) Before: . .
.
.
Decision making was based on inaccurate and historical data. It took several hours to collect and analyse data, this usually meant that in most cases it was not done at all or not done properly. Arguments about the validity of data caused friction between individuals. More often the organisation reacted to external stimuli, such as customer complaints, rather than internal stimuli due to the lack of confidence in the information available.
(2) After: .
.
Decisions can now be made quickly with confidence as the information is available in real-time or near real-time and the trends are visual on ‘‘active’’ SPC charts throughout the business. Information is accurate and reliable – because it is from a single source. This also eliminated animosity and friction between
Web enabled measurement systems 1281
IJOPM 22,11
individuals caused as a result of arguments over the validity of data – ‘‘we are managing by fact, using a single set of data’’. .
1282
It is easier to conduct root cause analysis to resolve problems and improve operations.
Impact on management style Most of the respondents have indicated that management decisions have become more proactive by selecting the ‘‘some improvement’’ option. They said that: (1) Before. Managers were reactive and unfocused because of duplicated information and low levels of confidence in the information. (2) After implementing MTR: .
.
Common view of transparent information, which allows managers to discuss and plan work to improve performance and then act on it. When things go wrong everybody knows the reason – it is annotated on the charts. Visibility like this makes us get on with our jobs rather than arguing about the data.
Effect on management attitude and behaviour Many people responded that the MTR system has a significant positive effect in many cases. In isolated cases some negative effects were mentioned, mainly related to manual data collection and sorting. They said that: (1) Before: .
. .
Easy not to be a ‘‘team player’’ and hide behind the poor data and so use a political approach to each situation. Most decisions were based on instinct and not on facts. There was a lot of pressure and wasted time before weekly management meetings, analysing data, understanding what went wrong and justifying why – ‘‘we used to try to justify the past’’.
(2) After: .
.
. .
Managing director used the MTR system to change the management style from reactive to proactive by insisting on all analysis to be based on statistical approach. Now there are no places to hide. This was difficult to get to grips with to start with. A lot of people felt very exposed. But the open management style helped a lot to overcome the fears. If you are not a ‘‘team player’’ it becomes very obvious very quickly. Now managers just turn up to the weekly management meetings and discuss what they are going to do rather than trying to explain what went wrong and why – ‘‘we are now managing the future’’.
Effect on dissemination of knowledge throughout the organisation Almost everybody responded that there is a significant positive effect. Previously, only a few people knew what was happening in the company, but now everybody knows what’s happening. They said that: (1) Before: . Knowledge was only disseminated through management briefings. . Consequently, some of the information and knowledge were retained at top levels without reaching operational levels. . Each function collected its own data independent from each other. (2) After: . All critical information and knowledge is completely accessible by all members of the organisation. . Support and operational staff (e.g. line operators, manufacturing engineers, quality engineers, etc.) use the information because they need to resolve problems. Management use the information because they need to make decisions, drive improvement and create focus. Business weaknesses highlighted by MTR system Half of the people responded that it highlighted some weaknesses. The other half responded that it highlighted many weaknesses. Here the main message was that before, people did not understand the effect of operational problems on business performance. Improvements were mainly driven by reactions to customer complaints. Now the cause and effect relationship is better understood and more visible, improvements are internally driven rather than being reactions to feedback from customers. Discussion The objective of this paper is to identify the management implications of IT supported performance measurement systems, because the current literature in performance measurement suggests that with IT support, performance measurement can be made less cumbersome, more dynamic and responsive, thus leading to a more proactive management style. One of the first questions asked related to the cost-benefit of such systems. At the conclusion of this research this was a question the researchers felt unsatisfied with because: Although it was fairly easy to establish the implementation costs, it proved difficult to establish the operational costs or saving of such systems (i.e. the marginal cost or saving of operating maintaining, updating and using such a system). As the MTR system became an inextricable part of the company’s processes and structures, it became increasingly difficult to separate the impact of the MTR system from other assets and activities (Willcocks and Lester, 1999). Another related question is, how are the benefits achieved in this company correlated with the total investment of implementing the MTR system? Even though the paper
Web enabled measurement systems 1283
IJOPM 22,11
1284
demonstrated that there are overall benefits of the MTR system, it failed to address whether the benefits are worth the MTR system investment. However, the management team’s opinion was that the business benefits enjoyed were largely attributable to the MTR system and that it far outweighed its costs. Figure 3 summarises the results of the feedback received from the selected cross-section of employees within AFE, providing qualitative answers to the rest of the questions posed earlier in the paper. The results of this research may be summarised as follows. The Webenabled performance measurement system, locally known as the MTR system, implemented in AFE plant resulted in significant benefits by: .
making performance information more transparent and visible;
.
improving accuracy, reliability and credibility of performance information;
.
creating awareness of issues and focus on critical problems;
.
creating an understanding of the cause and effect relationship between the business measures and operational measures.
Consequently, managers are: .
more confident with their decisions;
.
more proactive in their management style;
.
working more as a team than individuals;
.
more efficient as a team.
As a result, AFE demonstrated improvements in its bottom-line performance, which is measured through economic value added (EVA).
Figure 3. Summary of structured interviews
The researcher’s opinion is that the majority of the benefits gained are attributable not only to WePMS but also to other factors, which played a significant role in its success. These are: . Adoption of a framework (in this case IPMS Reference Model) to structure the performance measurement system. It is evident that creating a set of performance measures and publishing these on the Web would not have created the depth and breath of understanding evident in this case. . Adoption of Shewhart charts as a standard method of documenting performance information. In a process plant where SPC techniques are widely understood, this approach gained considerable support mainly due to familiarity and also due to visual and graphical nature of the charts. . Senior management commitment, was by far one of the key influences that led to the success of the MTR system. The managing director insists that all personnel, when they are talking to him on performance related issues, use the MTR system. He said: ‘‘I look at several charts several times every day, I ask questions to my managers and team leaders about them, I expect them to communicate to me using these charts’’. In our opinion it is his commitment that resulted in adoption of the MTR system as an everyday management tool. . Open and non-threatening management style from the very top of the organisation. One of the difficulties with the implementation of this system was that with the previous systems it was easy for managers to hide behind inaccurate and historical data. The difficulty was that as soon as the managers found out about the project they felt threatened and vulnerable. This feeling continued until the system was operational and they could see that the general manager was using the system to improve the business, and not to point fingers and apportion blame. The general manager’s continuous assurance and his sensitive approach throughout the development and implementation was a key factor for the success of this system. . Data collection and analysis was integrated into the business as part of one’s everyday job. Where possible, data collection was automated. In certain areas data collection and recording methods were changed, e.g. customer complaints are logged into a database instead of a book, allowing the MTR system to collect data directly from the database. These conclusions are consistent with Sauer (1993) who stated that every successful IT project has a new work practice supporting it. The discussion above makes it clear that providing just the hardware and software would not have resulted in these benefits. Other major influencing factors include changes in working practices, general managers’ commitment and an open and nonthreatening management style.
Web enabled measurement systems 1285
IJOPM 22,11
1286
Conclusions This paper provides empirical evidence that appropriately designed performance measurement systems, if supported through appropriate IT platforms, appropriately implemented will improve visibility, communications, teamwork, decision making and will lead to a more proactive management style. However, this conclusion is based on a single case study. The authors are confident that the benefits enjoyed by AFE are replicable in other organisations provided that the environmental conditions are similar. For example, it would be interesting to see what happens to the MTR system in AFE if another general manager with different management style and values was to take over the business. Therefore, more research is required to provide answers to the following questions: . How valid and generalisable are these results? .
.
What is the impact of different business environments on the performance of the WePMS? How does WePMS perform under different cultural environments?
References Beer, S. (1985), Diagnosing the System for Organisations, Wiley, Chichester. Begemann, C. and Bititci, U.S. (1999), ‘‘It platforms for performance measurement’’, client report. Bititci, U.S. (2000), ‘‘Dynamics of performance measurement systems’’, International Journal of Operations Management, Vol. 20 No. 6, pp. 692-704. Bititci, U.S. and Carrie, A.S. (1998), Integrated Performance Measurement Systems: Structures and Relationships, EPSRC Final Research Report, Grant No. GR/K 48174, Swindon. Bourne, M. and Neely, A. (2000), ‘‘Why performance measurement interventions succeed and fail’’, Proceedings of the 2nd International Conference on Performance Measurement, Cambridge, pp. 165-73. Burns, N. and Backhouse, C. (1998), Integrated Performance Measurement Systems – Behavioural Issues, EPSRC Final Research Report, Research Grant No. GR/K 48198, Swindon. Coleman, T. (1998), ‘‘Performance arts’’, InformationWeek, No. 21. Cross, K.F. and Lynch, R.L. (1988-1989), ‘‘The SMART way to define and sustain success’’, National Productivity Review, Vol. 9 No. 1. Dixon, J.R., Nanni, A.J. and Vollmann, T.E. (1990), The New Performance Challenge: Measuring Operations for World Class Competition, Dow Jones Irwin, Homewood, IL. Doumeingts, G., Clave, F. and Ducq, Y. (1995), ‘‘ECOGRAI – a method for designing and implementing performance measurement systems for industrial organisations’’, in Rolstadas, A. (Ed.), Benchmarking Theory and Practice, Chapman & Hall, London. Druker, P.E. (1990), ‘‘The emerging theory of manufacturing’’, Harvard Business Review, May/ June, pp. 94-102. EFQM (1998), Self-assessment Guidelines for Companies, European Foundation for Quality Management, Brussels, Belgium. Hudson, M., Bennet, J.P., Smart, A. and Bourne, M. (1999), ‘‘Performance measurement in planning and control in SME’s’’, in Mertins, K., Krause, O. and Schallock, B. (Eds), Global Production Management, Kluwer Academic Publishers, Dordrecht.
Johnson, H.T. and Kaplan, R.S. (1987), Relevance Lost – The Rise and Fall of Management Accounting, Harvard Business School Press, Boston, MA. Kaplan, R.S. (1990), Measures for Manufacturing Excellence, Harvard Business School Press, Boston, MA. Kaplan, R.S. and Norton, D.P. (1996), Translating Strategy into Action: The Balanced Scorecard, Harvard Business School Press, Boston, MA. Kehoe, D.F. and Little, D. (1998), Integrated Performance Measurement Systems – Information Systems, EPSRC Final Research Report, Research Grant No. GR/K 47818, Swindon. Kochhar, A., Kennerly, M. and Davies, A. (1996), Improving Your Business Through Effective Manufacturing Planning and Control, workbook produced by researchers at UMIST as part of an EPSRC Funded research programme. Krause, O. (1999), ‘‘Performance management’’, in Mertins, K., Krause, O. and Schallock, B. (Eds), Global Production Management, Kluwer Academic Publishers, Dordrecht. McNair, C.J. and Masconi, W. (1987), ‘‘Measuring performance in advanced manufacturing environment’’, Management Accounting, July. Neely, A. and Adams, C. (2001), ‘‘The performance prism perspective’’, Journal of Cost Management, January/February. Neely, A., Mills, J., Gregory, M., Richards, H., Platts, K. and Bourne, M. (1996), Getting the Measure of Your Business, University of Cambridge, Cambridge. Oliver, N. (1996), Design and Development Benchmarking, 5th Operations Strategy and Performance Measurement Workshop, Loughborough University, 8 May. Oliver Wight (1993), The Oliver Wight ABCD Check List, 4th ed., John Wiley & Sons, New York, NY. Russell, R. (1992), ‘‘The role of performance measurement in manufacturing excellence’’, BPICS Conference, Birmingham. Sauer, C. (1993), Why Information Systems Fail: A Case Study Approach, Alfred Waller, London. Willcocks, L.P. and Lester, S. (1999), Beyond the IT Productivity Paradox, John Wiley & Sons, Chichester.
Web enabled measurement systems 1287
The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/0144-3577.htm
IJOPM 22,11
The success and failure of performance measurement initiatives
1288
Perceptions of participating managers Mike Bourne and Andy Neely Centre for Business Performance, Cranfield School of Management, Cranfield, UK, and
Ken Platts and John Mills Centre for Strategy and Performance, Manufacturing Engineering, University of Cambridge, Cambridge, UK Keywords Performance measurement, Management, Implementation Abstract This paper investigates the success and failure of performance measurement system design interventions in ten companies. In each case, the senior management team was facilitated through a management process to redesign their performance measurement systems. Analysis of the initial pilot case studies suggested three important differences between companies that proceeded to implement the measures agreed during the process and those that did not. Post intervention semi-structured interviews with the directors and managers directly involved revealed two main perceived drivers of implementation and four perceived factors that block implementation. The results are of specific interest for performance measurement system implementation but have wider implications for our view of management commitment in change management.
Introduction With the balanced scorecard being cited by Harvard Business Review as one of the most important management tools of the last 75 years, performance measurement is currently attracting a great deal of interest among both industrialists and academics alike. However, although there are numerous balanced performance measurement frameworks (Keegan et al., 1989; Lynch and Cross, 1991; Fitzgerald et al., 1991; Kaplan and Norton, 1992; Neely et al., 2002) and management processes for the design of performance measurement systems (Bitton, 1990; Dixon et al., 1991; Kaplan and Norton, 1993, 1996; Neely et al., 1996; Bititci et al., 1998; Krause and Mertins, 1999), there has been less research into the success and failure of performance measurement initiatives. From industrial conferences (e.g. Business Intelligence, 2000), one may well come to the conclusion that there are few problems with implementing new performance measurement systems. There are certainly many success stories (e.g. Kaplan and Norton, 2000, the Mobile case), but there is now a growing literature addressing the difficulties of implementation and it is claimed by some that 70 per cent of performance measurement initiatives fail (McCunn, 1998). International Journal of Operations & Production Management, Vol. 22 No. 11, 2002, pp. 1288-1310. # MCB UP Limited, 0144-3577 DOI 10.1108/01443570210450329
Research for this paper was conducted during the research project – Manufacturing Strategy and Performance Measurement – which was sponsored by the CDP section of EPSRC under grant numbers GR/H21470 and GR/K53086.
Much of the existing literature is based on practitioner reflections and there have been few research studies of performance measurement implementation (Lewy and Du Mee, 1998; Bourne et al., 1999; Hudson et al., 2001 excepted). But drawing from this literature, and categorising them using Pettigrew et al.’s (1989) organisational context, development process and measurement content, the main reasons for success and failure are currently believed to be: (1) Contextual issues: . The need for a highly developed information system (Bierbusse and Siesfeld, 1997). . Time and expense required (Bierbusse and Siesfeld 1997; McCunn, 1998). . Lack of leadership and resistance to change (Hacker and Brotherton, 1998; Meekings, 1995). (2) Processual issues: . Vision and strategy were not actionable (Kaplan and Norton, 1996) as there were difficulties in evaluating the relative importance of measures and the problems of identifying true ‘‘drivers’’ (Bierbusse and Siesfeld, 1997; Schneiderman, 1999). . Strategy was not linked to resource allocation (Kaplan and Norton, 1996; Meekings, 1995). . Goals were negotiated rather than based on stakeholder requirements (Schneiderman, 1999). . State of the art improvement methods were not used (Schneiderman, 1999). . Striving for perfection undermined success (McCunn, 1998; Schneiderman, 1999). (3) Content issues: . Strategy was not linked to department, team and individual goals (Kaplan and Norton 1996; Bierbusse and Siesfeld, 1997; Schneiderman, 1999). . Large number of measures diluted the overall impact (Bierbusse and Siesfeld, 1997). . Metrics were too poorly defined (Schneiderman, 1999). . The need to quantify results in areas that are more qualitative in nature (Bierbusse and Siesfeld, 1997). If one reviews this list, the majority of the items are process and measurement content issues; the very issues the performance measurement design processes referenced above are specifically developed to address. This leaves the classic change management issues of leadership and resistance to change with the
Performance measurement initiatives 1289
IJOPM 22,11
1290
other contextual factors of time and expense and IT systems. Given that much is based on practioners’ reflections, further research is required. The change management literature provides general advice about how change should be implemented (e.g. Pugh, 1978; Beer et al., 1990; Duck, 1993, Eccles, 1994) and why change fails (e.g. Kotter, 1995) but there is a dearth of research based studies of performance measurement system implementations. As Frizelle’s (1991) study of studies showed, there are company general attributes, project general attributes and project specific attributes. The company and project general attributes (such as top management commitment, a perceived need for change and involving the end users) are well known attributes promoted in the change management literature. However, the project specific attributes for performance measurement are not. Therefore, this paper investigates the major factors that impact the success and failure of the implementation of a performance measurement system following action research interventions in ten manufacturing companies. All the interventions were undertaken using the same management process so that ‘‘process related factors’’ could be eliminated from the investigation. Although there was extensive case study data collected during six of the interventions, the results of the case and cross-case analysis raised more questions than they answered. Therefore, post intervention interviews were conducted with the 25 directors and managers who were directly involved in the performance measurement projects. The interviews focused on what the participating directors and managers perceived were the main reasons for success, the main reasons for failure and the difficulties encountered and overcome. An analysis of the interviews revealed that the directors and managers perceived that there were six main factors that influenced success and failure. These split into two main drivers of success and four barriers. The barriers to implementation also split into two categories, those that were insurmountable, and those that nearly all companies met, but some overcame and others did not. The paper will, therefore, make a contribution by giving us a better understanding of process success, clarifying and modifying our understanding of specific performance measurement project success factors and presenting one of the basic tenets of the change management literature in a new light. The paper will conclude by highlighting the insights and shortcomings of these findings and identifying the need for further research. Background All the companies involved in this research were medium sized (50-500 employees) manufacturing businesses, either privately-owned companies or subsidiaries of larger groups. The senior management team of each business (the managing director or general manager and his direct reports, see Table I) were facilitated through the same tried and tested performance measurement system design process (Neely et al., 1996). Each application took between five and six workshops to complete. The intended outcome was an agreed top level balanced scorecard for the business and this was achieved in eight out of the ten applications. In two applications the management team failed to complete
Progress Part 1. Part 2. Part 3. Part 4. Part 5.
Groups established Objectives agreed Measures designed Measures agreed Measures implemented
Senior team participating in process
Business
Ownership
B Machine Shop
Yes Yes No No No
MD, production director, commercial director and technical director, part 1 only Yes Yes Yes Yes No
Group manufacturing manager, site general manager and six local managers
Subsidiary of UK Subsidiary of Plc large US corporation Control Group central Instruments European machine shop
A Controls Ltd
Yes Yes Yes Yes No
Managing director, two directors and four managers
Subsidiary of large Japanese corporation Components manufacturer and supplier
Privately owned
Subsidiary of private group
F Seals
Yes Yes Yes Yes Yes
Yes Yes Yes Yes Yes
Yes Yes Yes Yes Yes
Manufacturers Manufacturers of Manufacturers of seals and of leak detection pumps and compression turbines equipment packings Site director, four Managing Managing director and four director and four BU general managers, and 11 directors managers managers
Venture capital backed
Company and given name C D E Components Plc Instruments Ltd Pumps Ltd
Performance measurement initiatives 1291
Table I. Summary of initial cases
IJOPM 22,11
1292
the process. The study was deliberately limited to the implementation of the top level measures (between 7 and 17 across the cases) and excluded investigations of the further cascading of supporting measures (although there was evidence of this occurring in all successful companies). Prior to the intervention, all the companies had been using financial performance measures as the basis of regular business level management reviews although some non-financial performance measures were in use in the main functions (e.g. sales, manufacturing). After the interventions, the eight companies completing the process populated all four of Kaplan and Norton’s (1992) scorecard perspectives. None of the scorecards was unduly biased to any one perspective although the innovation and learning perspective was consistently the least measured. The workshops were deliberately planned at three to four week intervals so that the development of the scorecard became part of every day business and not just a ‘‘weekend away’’ exercise. The teams completed most of the development during the workshops but the design of the measures (Neely et al., 1997) was conducted outside the main workshops by members of the senior management team and their direct reports. The measures so developed were presented back to the senior team as a whole and reviewed at the following workshop. The whole design process took between 15 and 26 weeks with the subsequent implementation in successful companies (from the end of the workshops to measures being displayed around the business) a further nine to 13 months (Bourne et al., 2000). The main commitment the businesses made to the project was management time, initially in attending the workshops and developing the measures, but latterly in coordinating the implementation work of others. All implementations involved in-house IT personnel and although different approaches were used to extract the data, all used standard spread sheet technology to record and present the final measures. No external IT resources were used in any of the interventions. Success in this research was defined as occurring when the management teams used the majority of the measures in the management of their business. The evidence to confirm this came from attending at least two board meetings, reviewing company reporting documentation and observing the measures being displayed around the business (in departments and in the canteen or other central meeting places). However, none of the companies who failed to progress their implementation denied the fact, so there was no dispute between the researchers and company personnel over whether the measures were implemented or not. In the next section, the pilot cases are described with the interviews in the following section. The pilot cases Earlier research (Bourne et al., 1999) investigated the success and failure of performance measurement interventions through action research case studies conducted in six manufacturing companies (hereafter referred to as the pilot cases).
During these interventions, the senior management team of the case study companies were facilitated through a performance measurement design process using a common tried and tested methodology (Neely et al., 1996). Data were collected before, during and after the interventions using a detailed case study protocol (Bourne et al., 1997) based on the prescriptions of Yin (1994) and Pettigrew et al.’s (1989) framework for research strategic change. Table I summarises the case companies involved and the progress towards implementation each achieved. The detailed case and cross-case analysis revealed three factors in the pattern of the data that differentiated the companies that successfully implemented the performance measures from the companies that did not. These were: (1) Purpose. At the start of the project, the senior manager or director of unsuccessful companies tended to express his purpose for undertaking the project in terms of improving their companies’ performance measures and performance measurement. The senior director of successful companies tended to express their purpose for undertaking the project in terms of managing the business better. (2) Structure. All the successful companies were privately owned. All the unsuccessful companies were part of larger publicly quoted groups. (3) Culture. All the successful companies had cultures that could be considered to be paternalistic. The culture of the unsuccessful companies was not described this way. These factors are discussed in turn: (1) Probably the most compelling of the three factors identified above is ‘‘purpose’’. It could be argued that if companies clearly saw performance measurement as a technique for improving the management of the business and moving the business forward, they would be more likely to go on and implement the measures. Companies that did not have this clarity of vision would not go on to implement the measures. (2) The fact that privately-owned companies implemented the measures and subsidiaries of larger groups did not is an observation, not an explanation. An explanation of how the ‘‘structure’’ impacted implementation of the measures requires an understanding of the interactions between the parent company and its subsidiary. In particular, the interactions which negatively influenced the implementation of the performance measures need to be identified. Although there are some indications of how parent company initiatives had a negative impact on the performance measurement implementation, there is insufficient evidence from these cases to make a clear statement of how this worked in practice. (3) Finally, although all the successful companies had cultures which could be considered to be paternalistic, this was not measured scientifically. The attribution of ‘‘paternalistic’’ to the culture of the companies was made from direct observation of the strategy formulation process. The
Performance measurement initiatives 1293
IJOPM 22,11
1294
attribution then appeared in the cross-case analysis. It can be argued that having a paternalistic culture could be beneficial for the implementation of performance measures. One would expect the fear of the consequences of measurement to be less, reducing the resistance to the implementation. However, considering the substantial time and effort invested in conducting these six case studies, the results were disappointing. It was, therefore, decided to follow up the pilot case studies with a series of semi-structured interviews to try to obtain some greater insights into why the interventions succeeded in certain circumstances and did not in others. The interview research Interviews in the six pilot case companies were undertaken, as this allowed the results from the case studies to be analysed in parallel with the perceptions of the managers and directors who had directly participated in the process workshops. However, there was an opportunity to interview managers and directors of four other companies who had also undertaken the process in the 12 months preceding the pilot cases. These included three companies in which the process was facilitated by one of their own employees and one company in which the process had been facilitated by the one of the paper’s authors as part of his training in using the process. These additional four companies were selfselecting as being the only other group of companies to whom the researcher had access and who had recently attempted to implement a performance measurement system using the Neely et al. (1996) process. Given the small population of potential interviewee’s and the dearth of our knowledge on performance measurement implementation problems, from evaluating the methodological options available (Mintzberg 1973) it was decided to adopt a grounded approach (Glaser and Straus, 1967) rather than use a formal questionnaire. Therefore, a semi-structured interview protocol was designed (and pre-tested) with the intention of gaining uninhibited responses to open ended questions concerning the reasons why implementation proceeded or not. However, there was also a concern that the interviewee may focus on one single factor and ignore other important factors. To avoid this problem and to provide some degree of consistency across interviews, a semi-structured approach was adopted, with the resulting interview format being as follows: . each interview started with a short series of open ended questions; . the responses to these open ended questions were probed to ensure that the interviewer understood the points being made and to gather specific examples; . the open ended questions were followed by questions focused on a prompt list of possible factors; . the responses to the prompt list were probed to ensure understanding. The prompt list was developed from the original research design and so focused on Pettigrew et al.’s (1989) dimensions of organisational context, performance measurement content and the process of designing and
developing the measures. As can be seen from Table II, these prompts were tailored to elicit the main reasons for continuing to implement the performance measures and the main reasons for not continuing. Successful companies were also asked to identify problems they had encountered but overcome. Data were collected during the interviews through the researcher making detailed notes as the interview was in progress. These notes were then written up within 24 hours of the interview and then later used as the basis for analysis. The analysis involved coding the responses given and using this summarised data to identify the main themes arising during the interviews. Because the interviews were in a semi-structured format, the interviewer took advantage of the discretion this allowed to follow up the answers given to the original questions. In most of the interviews undertaken with the companies that did not proceed to implement their performance measures, detailed probing of the responses given led to far greater insights into why the initiatives failed than given in the initial bland responses. These responses were gained by asking oblique additional questions which reflected back what the interviewer had revealed but questioned the completeness of the initial response. In all cases this elicited information of a more confidential nature and as a direct result the description of the four additional cases has had to be heavily disguised. The semi-structured interviews were conducted with directors and managers who had originally directly participated in the initial performance measurement design workshops. These interviews were conducted between nine and 18 months after the intervention so that it would be clear whether or not the measures designed and agreed during the process were implemented or not. The results of the interviews are presented in two groups: (1) The participating managers in the companies who successfully implemented their performance measures (Leak Detection Ltd, Pumps Ltd and Seals: cases D, E and F from the pilot cases above). (2) The participating managers in the companies who did not implement their performance measures (Machine Shop, Controls Ltd and Components plc: cases A, B and C from the pilot cases above) and in four other companies who had previously attempted the process (Food, Consumer Consumables Ltd, Models Ltd and Specialist Engineering: cases G, H, I and J). The successful companies Leak Detection Ltd, Pumps Ltd and Seals all progressed to implement the performance measures designed and agreed during the performance measurement workshops. The next section focuses on the reasons for continuing and the following section discusses the difficulties overcome.
Performance measurement initiatives 1295
Table II. Summary of prompted interview responses 4 5
We got good early results
The key people pushed the project along The project was given priority
The project was well managed and coordinated The facilitation brought out the key issues The process focused us on key points There was enthusiasm to continue after the facilitator left We were getting a lot out of the process for the time we took
Senior management were committed to doing this We had not got everything out of the project by then
2
People made time for the project
The facilitator left and the project petered out We did not think the results were worth the effort we put into the project
1 6
5
7
The project was badly managed or coordinated The facilitation didn’t meet up to our requirements The process was too cumbersome
There was a lack of management time We had problems implementing the measures/action plans we designed Key people left and the momentum was lost The project was overtaken by other events Senior management commitment to the project waned We got what we wanted from the project and stopped
This kind of initiative was never going to work here
2
5
8
4
6
This was exactly the kind of initiative we needed
Reasons for continuing from list
1296
1
1
2
1
3
3
1
2
The facilitator left and momentum was lost Some did not think the results were worth the effort we put into the project
Doing this kind of initiative was always going to be difficult in this company There was a lack of management time We had problems implementing the measures/action plans we designed Key people left and the momentum was lost The project was nearly overtaken by other events Senior management commitment to the project waned We got what we initially wanted from the project and there was pressure to stop The project was badly managed or coordinated The facilitation didn’t meet up to our requirements The process was difficult to use
Frequency Frequency of of response Reasons for not continuing from list response Difficulties overcome from list
1
1
2
5
3
3
Frequency of response
IJOPM 22,11
The successful companies’ reasons for continuing Analysis of the semi-structured interview results revealed five main themes as to why these companies progressed to implement the performance measures. These five themes (in order of frequency of citation) were: (1) The benefits of performance measurement (oded as ‘‘benefits’’). (2) Continued top management commitment (coded as ‘‘TMC’’). (3) Time and effort required (coded as ‘‘worth effort’’). (4) As a consequence of the activities of the internal or external facilitator (coded as ‘‘facilitator’’). (5) The juxtaposition of the performance measurement intervention with other projects (coded as ‘‘other projects’’). These themes are now addressed in turn. Benefits. When asked why they had continued, every interviewee commented on the benefits they perceived arising from performance measurement. For example: They are working, if we had not measured we wouldn’t have seen the improvement in order quality and on-time delivery. The actions taken were a result of having the measures (Managing Director, Leak Detection Ltd). Firstly, they [the measures] focused on issues which were important to the company and secondly they forced us to look at the measures and in particular the changes when they were not happening and being reflected in the measures (Managing Director, Pumps Ltd). Without it we wouldn’t be where we are and we’ve moved a hell of a long way (General Manager A, Seals Ltd).
From these responses, it appears that the managers involved believed that performance measurement was ‘‘a good thing’’ per se and that it was delivering business results. For example: Because we believe in it . . . I believe in this more than a lot of other things (Works Director, Pumps Ltd). There is a belief now in the management team . . . [and later] . . . it’s bringing home the bacon (Personnel Manager, Seals).
‘‘Benefits’’ was by far the most cited reason for continuing, being cited more times than all the other reasons summed together. TMC. Top management commitment (TMC) to the project was cited in all the cases as a reason for continuing. Given the attention top management commitment receives in the change management literature (Kotter, 1996; Frizelle, 1991), this result is not unexpected. However, it is interesting that it is cited far less frequently than ‘‘benefits’’. Worth effort. The time and effort required to design and implement the measures was raised in two companies (Pumps Ltd and Seals). Time and effort are usually reasons for not continuing (see difficulties overcome, below) but were raised here as the interviewees believed the results were worth the time and effort required. For example:
Performance measurement initiatives 1297
IJOPM 22,11
1298
We’re not just doing this for your benefit, Mike, we wouldn’t be doing this if we didn’t think it was worth it (Sales Director, Pumps Ltd). [We continued] partly due to the process, which was tortuous but got buy in (Site Director, Seals).
Further, performance measurement had continued while other important projects had not continued because time and effort was not available: People are exceptionally busy. The P&L [divisional profit and loss account project] has fallen by the wayside because of other pressures (General Manager B, Seals).
Minor themes. The role of the facilitator was cited in Leak Detection Ltd and Seals as were the influence of other projects in Seals, but because of their infrequency of citation, they have been included as minor themes. The successful companies’ difficulties overcome In the second part of the semi-structured interviews, interviewees were asked to discuss some of the problems they faced when designing and implementing the performance measurement systems and in particular the difficulties they had overcome. Analysis of the semi-structured interview results revealed four main themes which were cited across all three case companies. These four themes (in order of citation) were: (1) Difficulties with data access and the information technology systems (coded as ‘‘IT’’). (2) The time and effort required (coded as ‘‘effort’’). (3) Difficulties concerned with developing appropriate measures (coded as ‘‘measuring’’). (4) The personal consequences of performance measurement (coded as ‘‘consequences’’). In addition, interviewees in Seals also commented on the impact of the culture in different parts of the organisation (coded as ‘‘culture’’) and the political problems of dealing with head office as a supplier (coded as ‘‘politics’’). IT. Data access was a recurring problem in two companies (Leak Detection Ltd and Pumps Ltd). Both had relatively new computer systems. The comments captured reflect difficulties in data accuracy, data access and the time and effort required for programming the data extraction reports required. Many of the problems at Leak Detection Ltd were overcome by manual collection of the data, but transaction volumes were too high for this to be done successfully at Pumps Ltd. Therefore, at Pumps Ltd, the implementation of the measures was totally dependent on the programming being completed. Seals operated a much older computer system run on a main frame computer at their head office in the South of England. However, they had the advantage of being able to directly access the data they wanted and download it onto local PC-based spread sheets using a package called Power Play. IT was, therefore, a significantly smaller problem for Seals and was even cited as a reason for continuing by the site director.
Effort. Besides the time and effort required to access data from the IT systems, interviewees also commented on the time and effort required for performance measurement. Comments were made in general terms:
Performance measurement initiatives
Finding the time, there is never quite enough (Operations Manager, Seals Ltd). Time is a commodity which is incredibly precious, especially for something that is new and has not proved itself (General Manager A, Seals Ltd).
Also they were made in terms of setting up the system: Initially it took time to set the data up. I had to dig out a year’s worth of back data, a year’s worth of warranty returns for example. I had to find half an hour here and half an hour there to do it. Now it takes no time at all (Quality Manager, Leak Detection Ltd).
In addition they were made in terms of using the implemented measures: Now it takes time for the management meetings because I have to research the reasons for a good or bad performance and explain them (Works Director, Pumps Ltd).
Measuring. The difficulty of measuring was raised during interviews across all three cases: There is a problem with re-quotes, when they go out of validity, without this it is not a sensible measure. The conversion rate is key, but we need to know the accuracy. Have we lost the order or re-quoted (Sales Manager, Leak Detection Ltd). . . . had a great deal of difficulty with the measure of increasing the number of orders for special products (Sales Director, Pumps Ltd). . . . the [problem of the] rigorousness of what gets included in the subjective measures (Site Director, Seals).
These comments appear to illustrate that the managers have understood the importance of the definitions of the individual measures and how they were struggling to overcome these difficulties through implementing meaningful measures. Consequences. The personal consequences of being measured were occasionally raised during the interviews and are included as a theme as they were cited across all three cases. For example, consequences were cited by the site director at Seals: It’s perceived as a big stick, but that is their perception . . . it should be a good way of managing.
They were seen as one of the reasons behind the slow implementation of the measures in Pumps Ltd: . . . there was the problem . . . a reluctance to implement the measures (Managing Director, Pumps Ltd).
Much more severe reactions to the personal consequences of the implementation of performance measurement systems are described in later interviews with managers of companies that did not progress to implementation. The type of personal consequence found in these cases is probably best illustrated by Leak Detection Ltd’s sales manager in his comment:
1299
IJOPM 22,11
1300
Presenting it [the performance measure] to people, when you have a problem. What annoys me is the operations manager is saying why don’t you do this and that. I don’t want their input, what do they know about sales, I will ask if I want it. I sort of resent it, I do resent it.
Minor themes. Within one case, Seals, three minor themes were identified. Changing the culture was seen as one of the main benefits of implementing performance measurement (Thompson, 1997). The differences between the cultures of the business units and service departments was cited as a reason for the different rates of progress, with business units implementing the measures much more rapidly than the service functions. This could be interpreted as resistance to performance measurement because of the personal consequences of doing so, but it was never described as such. ‘‘Politics’’ also appears in this case. The head office site was a major supplier of raw materials to Seals and believed to be one of their poorest performing suppliers. Two of the general managers wished to raise the issue with head office as result of the performance measurement project, but the site director prevented them from doing so. Summary. From the initial analysis of the results of the semi-structured post process interviews in the successful companies, five reasons for continuing to implement the performance measures were identified and seven difficulties overcome. In the next section, the results from the unsuccessful companies are presented and discussed. The unsuccessful companies As previously described, the pilot case companies, Controls Ltd, Machine Shop and Components plc, all failed to implement the performance measures designed and agreed during the performance measurement workshops. In addition, access was available to four other companies who had previously attempted the process (Food, Consumer Consumables Ltd, Models Ltd and Special Engineering). All four of these companies had also failed to implement the performance measures. Analysis of the semi-structured interview results revealed seven main themes as to why these companies did not implement the performance measures. These seven themes (in order of frequency of citation) were: (1) Time and effort required (coded as ‘‘effort’’). (2) The personal consequences of implementing the performance measures (coded as ‘‘consequences’’). (3) The perceived lack of benefit from proceeding with performance measurement (coded as ‘‘benefits’’). (4) Difficulties with data access and the information technology systems (coded as ‘‘IT’’). (5) Continued top management commitment (coded as ‘‘TMC’’). (6) The impact of parent company activities and initiatives (coded as ‘‘parent company’’). (7) Problems with applying the process (coded as ‘‘process’’).
These are now addressed in turn. Effort. As found with the successful pilot company cases, finding the time and effort was a problem and it was the most frequently cited reason for not continuing in these seven cases. For example: We are an under-resourced low volume company and there was permanently a lack of time and resource for doing things like this (Commercial Manager, Special Engineering). Finding the people and the time to do it is the real problem (Director, Models Ltd).
But also finding the time and effort became linked to the perceived benefits of continuing with performance measurement. For example: . . . we weren’t sure what we were going to get out of it, time became a premium and we lost commitment (Operations Director, Controls Ltd). There is a lack of confidence in the performance measures, if we go through the time and effort of producing the performance measures will it actually do anything different at the end of the day (Director, Models Ltd).
Consequences. Here the personal consequences are raised in two different situations. First, there is the situation where the individual who has the authority to implement the performance measures takes a deliberate decision not to do so as he believes it will portray adverse performance of the business for which he has responsibility. For example: The ones [performance measures] we had got are the ones the corporate wanted us to have and for which we get the stars (UK Manufacturing Manager, Machine Shop). It is not just the problem of change, it’s exposing yourself. All the worms are under the stones and the performance measurement system lifts the stones off (Director, Models Ltd).
Second, there is the situation where those being measured were in fear of performance measurement (in the same sense that Deming, 1986, writes about the fear and is similarly raised in the context of performance measurement by Meekings, 1995): It is just a question of using the measures, we are not mature enough yet to talk about solving the problems, everybody sees the measures as a personal attack on themselves . . . We are a two excuse company, the first excuse is the figures are bollocks, the second excuse is shoot the messenger (Commercial Manager, Special Engineering). . . . maybe you ought to run the next project on how you stop the bully, especially when the bully is the boss (Internal Facilitator, Food).
In both instances, the consequences for the individual of performance measurement appeared to be a major factor in the measures not being implemented. Benefits. The benefits from performance measurement were cited as a reason for continuing by every interviewee from the successful case companies. In this group of unsuccessful companies, lack of perceived benefit was cited three times (Controls Ltd, Machine Shop and Models Ltd).
Performance measurement initiatives 1301
IJOPM 22,11
IT. As with the successful companies, access to data was cited by three interviewees who reached the stage of agreeing their performance measures (Components Plc, Machine Shop and Models Ltd). TMC. Top management commitment was seen to wane in a number of ways: .
1302
as there was a lack of commitment to implement the action plan (Special Engineering);
.
as benefits became unsure and time a premium (Controls Ltd);
.
as everyone had their own agenda (Components Plc).
Parent company. The influence of the parent company on the implementation of the performance measurement system was very strong in four cases. First, at Components and Machine Shop, the parent company influenced what was reported: For Ray, if that’s his key performance measure [market share] then they [Japanese parent company] will measure him on that . . . Very heavily driven by the Japanese measures – they drive us (Operations Manager, Components Plc). . . . the ones we had got are the ones the corporate wanted us to have . . . (UK Manufacturing Manger, Machine Shop).
Second, at Consumer Consumables Ltd and Components Plc, changes in strategy were decided at the level of the parent company and then imposed on the subsidiary. These changes in strategy by the parent company are changes which go to the heart of the rationale behind undertaking the performance measurement process and therefore it is probably not surprising that these companies did not succeed in implementing their measures. Process. On two occasions, the application of the process was considered a factor. In the first instance (Food), this occurred when the process was applied in a research organisation and the facilitator found adapting the process difficult. In the second instance (Controls Ltd), the operations director considered that the process was inappropriate for the managing director. The prompted responses The result from the prompted responses (see Table II) highlighted the importance of senior management commitment to the project and the influence of other events already identified from the analysis of the open questions, but they also provide one further insight. The responses from the successful companies to the ‘‘reasons for continuing’’ list and from the unsuccessful companies to the ‘‘reasons for not continuing’’ list were strikingly different. On the former, every reason for continuing was cited at least once. On the latter, with the exception of ‘‘senior management commitment to the project waned’’ and ‘‘the project was overtaken by other events’’ most other categories were not cited as reasons for the implementation not proceeding.
Interpretation of the interview results Starting with the prompted responses, the difference between the reasons for continuing list and reasons for not continuing is interpreted in two ways. First, it provides strong evidence that the application of the process was not a major factor influencing success and failure as project management, facilitation and the process were not frequently cited reasons for failure. Second, the responses supported the unprompted interview results. As the prompt list was added to ensure that interviewees considered contextual, process and content factors, this gives some comfort that no major factors were overlooked. With regards to the unprompted responses, Table III shows the coded results from both the successful and unsuccessful companies. Table III has been deliberately drawn to superimpose the reasons for continuing, cited by the successful companies, with the responses from the unsuccessful companies, where these reasons were lacking. Table III is interpreted below. Reviewing Table III, there appear to be two types of factors identified here: those factors that compel progress and those factors that block progress. Taking the factors in turn and beginning with the factors that compel progress: . Top management commitment to the project and the perceived benefits the project brought are cited as reasons for continuing by respondents in all the successful cases. In comparison, either lack of perceived benefit or senior management commitment waning were cited as reasons for not continuing in six other case companies. This suggests that these two factors are immensely important for success. . Worth the effort is interpreted as being related to the benefits and effort involved and therefore should be seen under those two headings. It is included here as it suggests that management make a trade-off between effort and benefit. . Facilitation relates to the comments about the support provided by both the internal and external facilitator. It can be interpreted as relating either to an example of action which demonstrated top management commitment, or to the need for the project to be properly managed. As these codes already exist, facilitation is not considered sufficiently important to be a separate factor and so is subsumed into TMC and process. . Other projects was raised only once and therefore seen as an incidental contextual factor. This suggests that the factors compelling progress can be reduced to two important factors, the benefit from undertaking the project and top management commitment. Some of the comments made suggest that the two are linked, but this needs further investigation. Turning to the blocking factors: . The effort required was cited as a reason for terminating the project in all but one of the unsuccessful companies but cited as a difficulty overcome in all the successful companies. This suggests that the effort required can be overcome under the right circumstances.
Performance measurement initiatives 1303
Table III. Summary of unprompted interview responses Yes Yes Yes Yes
Yes
Worth effort Facilitator Other projects
Difficulties overcome Effort IT Measuring Consequences Process Parent company
Yes
TMC (top management commitment)
Yes
Yes Yes Yes Yes
Yes
Yes
Yes
Yes Yes
Yes
Yes Yes Yes
Yes
Yes
No
No
Lack of benefits Lack of TMC
No
No
No
Lack of benefits
No
No
Lack of TMC
No No
No
No
No
No
No No
Lack of benefits
Unsuccessful companies Initial pilot cases Previous cohort cases A B C G H I
1304
Reasons for continuing Benefits
Successful companies Initial pilot cases D E F
No
No No
Lack of TMC
J
IJOPM 22,11
.
.
.
.
.
IT had a similar pattern of responses to effort and is interpreted in the same manner. Measurement problems were cited only by the companies who had reached the stage of implementation. These companies were struggling to develop meaningful measures and the interpretation of the comments made is that they were trying to ensure that the results of implementing specific measures were worth the effort of measuring. This suggests that companies reaching this stage will find measuring a difficulty, but not be blocked from continuing to implement the measures. Consequences of measuring were raised in both the successful and unsuccessful responses. Process was raised in two companies as a reason for not continuing. When combined with the prompted response, this suggests that a satisfactory application of the process (the point of entry, procedure, project management and participation (Platts, 1994)) is a necessary but insufficient factor in determining the successful implementation of the performance measures. The disruption from other parent company initiatives was cited as a reason for not continuing in three cases and was unobserved in the successful cases. This suggests that the impact of parent company interventions requires further investigation.
In summary, there appear to be four main blocking factors to implementation of the measures: (1) the effort required; (2) the ease of data accessibility through the IT systems; (3) the consequences of measurement; (4) being overtaken by new parent company initiatives. The problems of effort and IT systems have been demonstrably overcome in the successful cases and therefore are not considered fatal to projects if the right conditions exist (for example if there is sufficient senior management commitment). However, the consequences of measurement and parent company initiatives require further investigation. Discussion The conclusions from the six pilot research cases were that there were indications of three factors which differentiated between the successful and unsuccessful companies. These were: (1) purpose; (2) structure; (3) culture.
Performance measurement initiatives 1305
IJOPM 22,11
1306
These factors could be related to three issues raised during the post process interviews – top management commitment, parent company intervention and consequences of measurement: (1) The fact that the senior directors of successful companies tended to express their purpose for undertaking the project in terms of managing the business better may well be an indication that at the outset there was a greater level of top management commitment, as ‘‘managing the business better’’ can be considered a higher goal than simply ‘‘improving the measurement system’’. (2) The reason for structure being identified as a factor in the pilot cases was not understood. However, the interview results, which suggested that parent company interventions interrupted implementation, provide an explanation as to why structure appeared as an important factor. (3) At the end of the pilot cases it was argued that a paternalistic culture might well be beneficial for the implementation of performance measures as this would reduce the fear of measurement and therefore, the resistance to implementation. The fact that the consequence of measurement was raised as a major factor during the interviews strongly supports this argument. Therefore, the semi-structured interviews provide some explanation of the results obtained from the cross-case analysis, and the results of the cross-case analysis provide evidence of the impact of three of the factors found from an analysis of the interviews. In addition to the situation where the findings from case studies and interviews coincided, the semi-structured interviews highlighted three other main factors that influenced the success or failure of implementation. These are now discussed in turn: (1) The perceived benefit of performance measurement should have been identified during the post project interviews in the pilot cases, although these interviews focused more on objective rather than subjective factors that influenced implementation. This may explain why the benefits were missed, but this could also have resulted from the fact that the benefits did not occur until much later – when the measures had been fully implemented. The later explanation is the more plausible and this suggests that the post project interviews were conducted too early to capture the perceived benefits of the project. (2) The IT hurdle was observed in the pilot cases but not consistently enough to be cited as a major difference in the cross-case analysis. The interviews raised the importance of this factor. (3) The effort required largely went unobserved. From the nature of the interventions undertaken, it was mainly unobservable and, as it was not a factor identified in the academic frameworks used to create the case study protocol, data were not collected on the effort required.
The use of multiple methodologies in this research highlights some of the additional insights that can be gained. In particular it suggests that the benefits from implementing a performance measurement system occur very late in the process and highlights the ‘‘effort’’ required, a factor not previously identified as important.
Performance measurement initiatives
Conclusions There are many studies of implementation, but it is exceptionally rare for a study to research the implementation of the same artefact in multiple organisations using a common and moderated process approach. Although this approach is open to criticism, it does control for variables that are not controlled in other studies and so provides different insights. This paper makes four contributions, as follows: (1) It qualifies Platts’ (1994) findings on process success. (2) It demonstrates that by using a structured performance measurement design process approach many of the implementation problems highlighted in the literature can be overcome. (3) It identifies the major performance measurement project specific factors influencing implementation. (4) It suggests that top management commitment changes during the project, a factor implicit in, but ignored by, much of the change management literature.
1307
These are now discussed in turn. First, Platts (1994) in his research into the application of management processes for the development of manufacturing strategy identified four factors associated with the application of the process which appeared to be significant. These were: point of entry (how the introduction and launch was handled), participation (who was involved), project management and procedure (the tools used in the process itself). This research supports his findings that these factors are necessary, but suggests that in themselves they may not be sufficient for successful implementation of performance measurement systems. Other non-process factors are also important. Second, many of the factors causing problems for implementation highlighted in the introduction to this paper (e.g. strategy and vision not actionable and measures poorly defined) could be attributed to poor design process. The fact that none of these issues was raised as an impediment to implementation by this research suggests that a well designed and executed process can overcome these implementation issues. Third, given that a suitable management process is correctly applied, the findings from this research proposes that there are four important performance measurement project specific implementation factors. These are: (1) The effort required for implementation.
IJOPM 22,11
1308
(2) The ease of data accessibility through the IT systems. (3) The consequences of performance measurement. (4) The project being overtaken by parent company initiatives. The results also show that problems with the IT systems and the effort required can be overcome. This suggests that these two blockers are simply hurdles to implementation rather than factors which completely stop the project. This work confirms three of the issues thought to be important from practioners’ reflections, but qualifies our understanding by differentiating between hurdles and substantial blockers. The impact of parent company initiatives is a significant factor not previously recognised in the literature. Fourth, TMC is widely taken as a major factor influencing the success and failure of project implementations (Frizelle, 1991). In this study, the interview results identified top management commitment and the benefits from performance measurement as being perceived as the two main factors which drove implementation. But, it must be remembered that the interviews were all conducted some time after the process was completed. Many of the managers were therefore reflecting on the perceived benefits of the performance measures after they were implemented. This raises the question: ‘‘What was supporting the performance measurement initiative through to implementation before the benefits occurred?’’ The results of this research suggest that management commitment could be influenced by the change in balance between the expected benefits of the intervention and the expected effort required to implement the performance measures. Therefore, management commitment may not be static, but change over time as the intervention unfolds. Careful reading of the change management literature alludes to this occurring (e.g. creating a guiding coalition (Beer et al., 1990; Goss et al., 1993; Kotter, 1996) and establishing the need for change (Pugh, 1978)) but even the processual approaches (Lewin, 1947; Dawson, 1996; Burnes, 1997) do not consider how TMC can be influenced over a project’s life. If management commitment is dynamic and is a dependant not an independent variable, then change management should be looked at in a new light. We would go further and suggest that the insights gained in this study would not have been raised by a ‘‘scientifically’’ designed questionnaire administered to a representative sample of managers. The semi-structured approach uncovered the ‘‘non-standard’’ responses as a direct result of probing and using knowledge from the original case studies. The conclusion that top management commitment is dynamic is not explicitly addressed in the change management literature and this may be the result of researchers restricting their methodologies. This is now the subject of further research using longitudinal case studies and data collection methods designed to track changes in TMC and the factors that influence these changes.
References Beer, M., Eisenstat, R.A. and Spector, B. (1990), Harvard Business Review, November/December, pp. 158-66. Bierbusse, P. and Siesfeld, T. (1997), ‘‘Measures that matter’’, Journal of Strategic Performance Measurement, Vol. 1 No. 2, pp. 6-11. Bititci, U., Carrie, A. and Turner, T. (1998), ‘‘Diagnosing the integrity of your performance measurement system’’, Control, April, pp. 9-13. Bitton, M., (1990), ‘‘Me´thode de conception et d’implantation de syste`mes de measure de performances pour organisations industrielles’’, the`se d’ automatique, Universite´ de Bordeaux I, Bordeaux. Bourne, M.C.S., Mills, J.F., Bicheno, J., Hamblin, D.J., Wilcox, M., Neely, A.D. and Platts, K.W. (1999), ‘‘Performance measurement system design: testing a process approach in manufacturing companies’’, International Journal of Business Performance Measurement, Vol. 1 No. 2, pp. 154-70. Bourne, M.C.S., Mills, J.F., Richards, A.H., Neely, A.D., Platts, K.W., Gregory, M.J., Barnes, D.L., Bicheno, J.R., Gill, R., Hamblin, D.J., Rowbotham, F. and Wilcox, M. (1997), ‘‘Manufacturing strategy and performance measurement research design’’, Manufacturing Systems Group working paper, University of Cambridge, Cambridge. Bourne, M.C.S., Mills, J.F., Wilcox, M., Neely, A.D. and Platts, K.W. (2000), ‘‘Designing, implementing and updating performance measurement systems’’, International Journal of Production & Operations Management, Vol. 20 No. 7, pp. 754-71. Burnes, B. (1997), ‘‘Organizational choice and organizational change’’, Management Decision, Vol. 35 No. 10, pp. 753-9. Business Intelligence (2000), Business Intelligence Conference, chaired by Kaplan, R.S. and Norton, D.P., 3-5 May, London. Dawson, P. (1996), ‘‘Beyond conventional change models: a processual perspective’’, Asia Pacific Journal of Human Resources, Vol. 34 No. 2, pp. 57-71. Deming, W.E. (1986), Out of the Crisis: Quality, Productivity and Competitive Position, Cambridge University Press, Cambridge. Dixon, J.R., Nanni, A.J. and Vollmann, T.E. (1991), ‘‘An instrument for investigating the match between manufacturing strategy and performance measures’’, working paper, Boston University, Boston, MA. Duck, J.P. (1993), ‘‘Managing change: the art of balancing’’, Harvard Business Review, November/ December. Eccles, A. (1994), Succeeding with Change: Implementing Action-driven Strategies, McGraw-Hill, London. Fitzgerald, L., Johnston, R., Brignall, T.J., Silvestro, R. and Voss, C. (1991), Performance Measurement in Service Businesses, The Chartered Institute of Management Accountants, London. Frizelle, G.D.F.M. (1991), ‘‘Deriving a methodology for implementing CAPM systems’’, International Journal of Production Management, Vol. 11 No. 7, pp. 6-26. Glaser, B.G. and Straus, A.L. (1967), The Discovery of Grounded Theory: Strategies for Qualitative Research, Aldine, Chicago, IL. Goss, T., Pascale, R. and Athos, A. (1993), ‘‘The reinvention roller coaster: risking the present for a powerful future’’, Harvard Business Review, November/December. Hacker, M.E. and Brotherton, P.A. (1998), ‘‘Designing and installing effective performance measurement systems’’, IIIE Solutions, Vol. 30 No. 8, pp. 18-23. Hudson, M., Stuart, P.A. and Bourne, M.C.S. (2001), ‘‘Theory and practice in SME performance measurement systems’’, International Journal of Operations & Production Management, Vol. 21 No. 8, pp. 1095-115.
Performance measurement initiatives 1309
IJOPM 22,11
1310
Kaplan, R.S. and Norton, D.P. (1992), ‘‘The balanced scorecard – measures that drive performance’’, Harvard Business Review, January/February, pp. 71-9. Kaplan, R.S. and Norton, D.P. (1993), ‘‘Putting the balanced scorecard to work’’, Harvard Business Review, September/October, pp. 134-47. Kaplan, R.S. and Norton, D.P. (1996), ‘‘Using the balanced scorecard as a strategic management system’’, Harvard Business Review, January/February, pp. 75-85. Kaplan, R.S. and Norton, D.P. (2000), The Strategy Focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment, Harvard Business School Press, Boston, MA. Keegan, D.P., Eiler, R.G. and Jones, C.R. (1989), ‘‘Are your performance measures obsolete?’’, Management Accounting, June, pp. 45-50. Kotter, J.P. (1995), ‘‘Leading change: why transformation efforts fail’’, Harvard Business Review, Vol. 73 No. 2, pp. 59-67. Kotter, J.P. (1996), Leading Change, Harvard Business School Press, Boston, MA. Krause, O. and Mertins, K. (1999), ‘‘Performance management’’, in Mertins, K., Krause, O. and Schallock (Eds), Global Production Management, Proceedings of the IFIP WG5.7 International Conference on Advances in Production Management Systems, September. Lewin, K. (1947), ‘‘Frontiers in group dynamics’’, Human Relations, Vol. 1 No. 1, pp. 5-47. Lewy and Du Mee (1998), ‘‘The ten commandments of balanced scorecard implementation’’, Management Control and Accounting, April. Lynch, R.L. and Cross, K.F. (1991), Measure up – The Essential Guide to Measuring Business Performance, Mandarin, London. McCunn, P. (1998) ‘‘The balanced scorecard: the eleventh commandment’’, Management Accounting, December, pp. 34-6. Meekings, A. (1995), ‘‘Unlocking the potential of performance measurement: a guide to practical implementation’’, Public Money & Management, October-December, pp. 1-8. Mintzberg, H. (1973), The Nature of Managerial Work, Harper & Row, New York, NY. Neely, A.D., Adams, C. and Kennerley, M. (2002), The Performance Prism, The Scorecard for Measuring and Managing Business Success, FT Prentice-Hall, London. Neely, A.D., Mills, J.F., Gregory, M.J., Richards, A.H., Platts, K.W. and Bourne, M.C.S. (1996), Getting the Measure of Your Business, Findlay, London. Neely, A.D., Richards, A.H., Mills, J.F., Platts, K.W. and Bourne, M.C.S. (1997), ‘‘Designing performance measures: a structured approach’’, International Journal of Operations & Production Management, Vol. 17 No. 11, pp. 1131-52. Pettigrew, A., Whipp, R. and Rosenfield, R. (1989), ‘‘Competitiveness and the management of strategic change processes’’, in Francis, A. and Tharakan, P.K.M. (Eds), The Competitiveness of European Industry: Country Policies and Company Strategies, Routledge, London. Platts, K.W. (1994), ‘‘Characteristics of methodologies for manufacturing strategy formulation’’, Computer Integrated Manufacturing Systems, Vol. 7 No. 2, pp. 93-9. Pugh, D. (1978), ‘‘Understanding and managing organisational change’’, London Business School Journal, Vol. 3 No. 2, pp. 29-34. Schneiderman, A. (1999), ‘‘Why balanced scorecards fail’’, Journal of Strategic Performance Measurement, special edition, pp. 6-11. Thompson, M. (1997), ‘‘Getting the measure of your business: the experience of James Walker’’, Presentation to the local branch of the Institute of Management, 12 November, Whitehaven. Yin, R.K. (1994), Case Study Research, Design and Methods, 2nd ed., Sage Publications, Thousand Oaks, CA.
