Evaluating the Impact of Leadership Development
Kelly M. Hannum and Jennifer W. Martineau
ffirs.indd ix
2/1/08 4:04:10 PM
About This Book Why is this topic important? No matter how effectiveness is defined, there is a constant drive for efficiency, productivity, and stewardship of resources. This drive pushes commercial and not-for-profit organizations to weigh options, balance tradeoffs, and make better decisions about internal processes, customer or client relations, product releases and distribution, new programs and services, and other issues. An evaluation system provides a logical and practical framework for collecting, assimilating, and communicating information. The approach to evaluation we present in this book can be applied in a variety of contexts, but our focus is the evaluation of leadership development initiatives. Although many professionals do difficult and excellent work in and with organizations and communities to build leadership capacity, tools for evaluating leadership development are not as readily available.
What can you achieve with this book? Our experience evaluating leadership development initiatives has given us specific ideas for how that work should be conceived and conducted: that it should be participatory, be integrated with initiative design, and enhance learning. Using a participatory approach means involving key stakeholders—people who are, or will be, affected by the initiative being evaluated and/or by the outcomes of the evaluation itself. This ensures that the evaluation will take into account their needs and perspectives and will make the process and results more meaningful and useful. When evaluation work is integrated into the design and implementation of an initiative, both the initiative and the evaluation are more effective, and learning can result. That learning is reflected in increased knowledge regarding barriers to and facilitators of change. In identifying how and where an initiative has been successful (or not), stakeholders can learn what processes, structures, areas, or systems support change as a result of the development initiatives, as well as what may need attention to be most effective. Evaluation helps a variety of people and groups of people meet several goals. For example, it helps them make informed decisions about how to improve development initiatives and examine the degree to which development goals have been accomplished and what work remains to be done. If development outcomes aren’t met, evaluation helps people understand why outcomes were not met, learn from their experiences, and apply those lessons to improving individual, team, organizational, and community effectiveness in terms that make sense for their context.
ffirs.indd i
2/1/08 4:04:04 PM
How is this book organized? The material in this book is divided into four chapters. The first helps you understand the evaluation function and process, and the second explains how to focus an evaluation. The third chapter deals with designing and conducting an evaluation, and the fourth explains how to communicate and use evaluation findings. Finally, we have included a list of evaluation resources: publications, e-mail and discussion groups, professional organizations, and websites.
ffirs.indd ii
2/1/08 4:04:05 PM
About Pfeiffer Pfeiffer serves the professional development and hands-on resource needs of training and human resource practitioners and gives them products to do their jobs better. We deliver proven ideas and solutions from experts in HR development and HR management, and we offer effective and customizable tools to improve workplace performance. From novice to seasoned professional, Pfeiffer is the source you can trust to make yourself and your organization more successful.
Essential Knowledge Pfeiffer produces insightful, practical, and comprehensive materials on topics that matter the most to training and HR professionals. Our Essential Knowledge resources translate the expertise of seasoned professionals into practical, how-to guidance on critical workplace issues and problems. These resources are supported by case studies, worksheets, and job aids and are frequently supplemented with CD-ROMs, websites, and other means of making the content easier to read, understand, and use.
Essential Tools Pfeiffer’s Essential Tools resources save time and expense by offering proven, ready-to-use materials—including exercises, activities, games, instruments, and assessments—for use during a training or team-learning event. These resources are frequently offered in looseleaf or CD-ROM format to facilitate copying and customization of the material. Pfeiffer also recognizes the remarkable power of new technologies in expanding the reach and effectiveness of training. While e-hype has often created whizbang solutions in search of a problem, we are dedicated to bringing convenience and enhancements to proven training solutions. All our e-tools comply with rigorous functionality standards. The most appropriate technology wrapped around essential content yields the perfect solution for today’s on-the-go trainers and human resource professionals.
w w w. p f e i f f e r. c o m
ffirs.indd iii
Essential resources for training and HR professionals
2/1/08 4:04:05 PM
A BO U T T H E C E N T E R F O R C R E AT I V E L E A D E R S H I P
T
he Center for Creative Leadership (CCL) is a top-ranked, global provider of executive education that develops better leaders through its exclusive focus on leadership education and research. Founded in 1970 as a nonprofit, educational institution, CCL helps clients worldwide cultivate creative leadership—the capacity to achieve more than imagined by thinking and acting beyond boundaries— through an array of programs, products, and other services.
Ranked in the top ten in the Financial Times annual executive education survey, CCL is headquartered in Greensboro, North Carolina, with campuses in Colorado Springs, Colorado; San Diego, California; Brussels, Belgium; and Singapore. Supported by more than five hundred faculty members and staff, it works annually with more than twenty thousand leaders and three thousand organizations. In addition, sixteen Network Associates around the world offer selected CCL programs and assessments. CCL draws strength from its nonprofit status and educational mission, which provide unusual flexibility in a world where quarterly profits often drive thinking and direction. It has the freedom to be objective, wary of short-term trends, and motivated foremost by its mission—hence our substantial and sustained investment in leadership research. Although CCL’s work is always grounded in a strong
v
ffirs.indd v
2/1/08 4:04:08 PM
foundation of research, it focuses on achieving a beneficial impact in the real world. Its efforts are geared to be practical and action oriented, helping leaders and their organizations more effectively achieve their goals and vision. The desire to transform learning and ideas into action provides the impetus for CCL’s programs, assessments, publications, and services.
CAPABILITIES CCL’s activities encompass leadership education, knowledge generation and dissemination, and building a community centered on leadership. CCL is broadly recognized for excellence in executive education, leadership development, and innovation by sources such as BusinessWeek, Financial Times, The New York Times, and The Wall Street Journal. OPEN-ENROLLMENT PROGRAMS Fourteen open-enrollment courses are designed for leaders at all levels, as well as people responsible for leadership development and training at their organizations. This portfolio offers distinct choices for participants seeking a particular learning environment or type of experience. Some programs are structured specifically around small group activities, discussion, and personal reflection, while others offer hands-on opportunities through business simulations, artistic exploration, team-building exercises, and new-skills practice. Many of these programs offer private one-on-one sessions with a feedback coach. For a complete listing of programs, visit http://www.ccl.org/programs. CUSTOMIZED PROGRAMS CCL develops tailored educational solutions for more than one hundred client organizations around the world each year. Through this applied practice, CCL structures and delivers programs focused on specific leadership development needs within the context of defined organizational challenges, including innovation, the merging of cultures, and the development of a broader pool of leaders. The objective is to help organizations develop, within their own cultures, the leadership capacity they need to address challenges as they emerge. Program details are available online at http://www.ccl.org/custom.
vi
ffirs.indd vi
About the Center for Creative Leadership
2/1/08 4:04:09 PM
COACHING CCL’s suite of coaching services is designed to help leaders maintain a sustained focus and generate increased momentum toward achieving their goals. These coaching alternatives vary in depth and duration and serve a variety of needs, from helping an executive sort through career and life issues to working with an organization to integrate coaching into its internal development process. Our coaching offerings, which can supplement program attendance or be customized for specific individual or team needs, are based on our ACS model of assessment, challenge, and support. Learn more about CCL’s coaching services at http://www.ccl.org/coaching. ASSESSMENT AND DEVELOPMENT RESOURCES CCL pioneered 360-degree feedback and believes that assessment provides a solid foundation for learning, growth, and transformation and that development truly happens when an individual recognizes the need to change. CCL offers a broad selection of assessment tools, online resources, and simulations that can help individuals, teams, and organizations increase their self-awareness, facilitate their own learning, enable their development, and enhance their effectiveness. CCL’s assessments are profiled at http://www.ccl.org/assessments. PUBLICATIONS The theoretical foundation for many of our programs, as well as the results of CCL’s extensive and often groundbreaking research, can be found in the scores of publications issued by CCL Press and through the center’s alliance with JosseyBass, a Wiley imprint. Among these are landmark works, such as Breaking the Glass Ceiling and The Lessons of Experience, as well as quick-read guidebooks focused on core aspects of leadership. CCL publications provide insights and practical advice to help individuals become more effective leaders, develop leadership training within organizations, address issues of change and diversity, and build the systems and strategies that advance leadership collectively at the institutional level. A complete listing of CCL publications is available at http://www.ccl.org/ publications.
About the Center for Creative Leadership
ffirs.indd vii
vii
2/1/08 4:04:10 PM
LEADERSHIP COMMUNITY To ensure that the Center’s work remains focused, relevant, and important to the individuals and organizations it serves, CCL maintains a host of networks, councils, and learning and virtual communities that bring together alumni, donors, faculty, practicing leaders, and thought leaders from around the globe. CCL also forges relationships and alliances with individuals, organizations, and associations that share its values and mission. The energy, insights, and support from these relationships help shape and sustain CCL’s educational and research practices and provide its clients with an added measure of motivation and inspiration as they continue their lifelong commitment to leadership and learning. To learn more, visit http://www.ccl.org/community. RESEARCH CCL’s portfolio of programs, products, and services is built on a solid foundation of behavioral science research. The role of research at CCL is to advance the understanding of leadership and to transform learning into practical tools for participants and clients. CCL’s research is the hub of a cycle that transforms knowledge into applications and applications into knowledge, thereby illuminating the way organizations think about and enact leadership and leader development. Find out more about current research initiatives at http://www.ccl.org/ research. For additional information about CCL, please visit http://www.ccl.org or call Client Services at (336) 545-2810.
viii
ffirs.indd viii
About the Center for Creative Leadership
2/1/08 4:04:10 PM
Evaluating the Impact of Leadership Development
Kelly M. Hannum and Jennifer W. Martineau
ffirs.indd ix
2/1/08 4:04:10 PM
Copyright © 2008 by Center for Creative Leadership Published by Pfeiffer A Wiley Imprint 989 Market Street, San Francisco, CA 94103-1741 www.pfeiffer.com Except as specifically noted below, no part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at http://www. wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Certain pages from this book are designed for use in a group setting and may be reproduced for educational/training activities. These pages are designated by the appearance of the following copyright notice at the foot of the page: Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com This notice must appear on all reproductions as printed. This free permission is limited to the paper reproduction of such materials for educational/training events. It does not allow for systematic or large-scale reproduction or distribution (more than 100 copies per page, per year), electronic reproduction, or inclusion in any publications offered for sale or used for commercial purposes—none of which may be done without prior written permission of the Publisher. Readers should be aware that Internet websites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read. For additional copies/bulk purchases of this book in the U.S. please contact 800-274-4434. Pfeiffer books and products are available through most bookstores. To contact Pfeiffer directly call our Customer Care Department within the U.S. at 800-274-4434, outside the U.S. at 317-572-3985, fax 317-572-4002, or visit www.pfeiffer.com. Pfeiffer also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Hannum, Kelly. Evaluating the impact of leadership development / Kelly M. Hannum and Jennifer W. Martineau. p. cm. Martineau’s name appears first on the earlier ed. published by Center for Creative Leadership in 2004. Includes bibliographical references and index. ISBN 978-0-470-18410-3 (pbk.) 1. Leadership—Evaluation. 2. Executives—Training of—Evaluation. I. Martineau, Jennifer. II. Title. HD57.7.M39257 2008 658.4'092—dc22 2007049547 Acquiring Editor: Lisa Shannon Director of Development: Kathleen Dolan Davies Marketing Manager: Brian Grimm Manufacturing Supervisor: Becky Morgan
Editor: Rebecca Taff Assistant Editor: Marisa Kelley Production Editor: Michael Kay
Printed in the United States of America Printing 10 9 8 7 6 5 4 3 2 1
ffirs.indd x
2/1/08 4:04:11 PM
C ON T E N T S
Preface Introduction
xiii xvii
C H A P T E R O N E : Understanding the Evaluation Function and Process CHAPTER TWO:
Focusing the Evaluation
CHAPTER THREE:
Designing and Conducting the Evaluation
1 7 37
Communicating and Using Evaluation Findings
101
References Recommended Resources Index About the Authors
115 117 129 137
CHAPTER FOUR:
xi
ftoc.indd xi
2/1/08 4:03:37 PM
ftoc.indd xii
2/1/08 4:03:38 PM
P R E FA C E
T
he Center for Creative Leadership (CCL) has conducted evaluations of its programs, products, and services since the early 1970s. It evaluates new and revised programs for quality and to answer specific questions (such as the point at which specific behaviors begin to show meaningful change). CCL also evaluates client-specific initiatives to determine the impact of leadership development on individuals, teams, organizations, and their communities.
CCL’s mission is to advance the understanding, practice, and development of leadership for the benefit of society worldwide. Therefore, we are intentionally sharing what we have learned with a broad range of individuals in the practicing fields of leadership development and evaluation, as well as academic fields such as psychology, education, and sociology, and other fields in which leadership development is increasingly important. In our evaluation work we use many tools that we have developed over the years by drawing on best practices and research. An appropriate evaluation approach depends on the expected outcomes for and the context of a development initiative. It’s our expectation and our hope that this book will add to the available information about evaluation tools targeted specifically to leadership development initiatives. We envisioned this publication as a way to educate and assist others in the process of evaluating leadership development. Therefore, we have combined basic
xiii
fpref.indd xiii
2/1/08 4:02:35 PM
education about evaluation with examples of evaluation design processes and tools specific to leadership development. This information gives individuals with or without evaluation training, but with some behavioral sciences knowledge or experience in the organizational development field, an understanding of what it takes to design, implement, report on, and use evaluation studies regarding the impact of leadership development on individuals, groups, organizations, and communities. While reading this book will not make you an expert evaluator, we have tried to present a broad array of tools and processes from which readers can choose when designing and conducting their own evaluations. There are many very good resources available for readers wishing to learn more about how to evaluate in general (for example, Davidson, 2005; Fitzpatrick, Worthen, & Sanders, 2004; Preskill & Torres, 1999). The framework we use in this book is based on that shared by Preskill and Torres (1999) and is flexible enough to meet many needs. For example, evaluators can use the processes described to build evaluation into a development intervention so that the evaluation becomes an integrated part of the intervention while at the same time determining its progress and success. Even if an evaluation is requested after an intervention has been completed, the tools and processes in this book can help evaluators design useful and relevant evaluations. We envisioned an audience for this book comprising a broad group of people: those on the front lines of leadership development who want to enhance their practice and/or gather evidence to demonstrate the value of their work. This group can include human resources managers, consultants, nonprofit groups that conduct leadership development training, foundations that fund leadership development, scholars, managers, employees, and volunteers with leadership development responsibilities. Much of our work is collaborative, and thus there are more people who have contributed to this work than we could possibly name. However, there are a few individuals without whose contribution this book would not have been a reality. Bart Craig, Chuck Palus, Hallie Preskill, Sharon Rogolsky, and Ellen Van Velsor—we owe a great deal to you for “playing along with us” at various times on this work. This is as much a product of your thinking and practice as it is of ours. We especially want to thank our CCL Evaluation colleagues: Jessica Baltes, Judith Steed, Gina Hernez-Broome, Tracy Patterson, Emily Hoole, and Meena Wilson. Many of the examples used in the book are directly a result of their work.
xiv
fpref.indd xiv
Evaluating the Impact of Leadership Development
2/1/08 4:02:36 PM
We are also grateful to our reviewers and the editorial staff, who provided us with additional perspectives that allowed us to create a better final version. We extend our thanks to Joanne Ferguson, Sylvia Graham-Shoulberg, Amy Gullickson, Karen Lewis, Brenda McManigle, Tracy Patterson, Hallie Preskill, Peter Scisco, Judith Steed, Ellen Van Velsor, Wynne Whyman, and Meena Wilson. Laurie Merritt played a central role by keeping us on track throughout the process. Susan Clark helped us a great deal in updating information for this revised version. Finally, and most important, our ability to have the experiences necessary to create this book rests on the support, encouragement, and understanding of our family and friends. Jennifer especially thanks her parents, Herbert and Ellen Wells, for placing a high value on education and hard work, and her husband and children—Jim, Sarah, Christopher, and Grace Martineau—for their undying patience with her love of her work and for adding so much joy to her life. Kelly would like to thank her parents, Elizabeth and Wallace Hannum; her almost but not quite legal stepmom, Nancy Kiplinger; and her siblings—Rebecca Rogers, Bryan Rogers, and Wallace Hannum—for their love and support. She also thanks Keith and Dylan Erikson for bringing a lot of fun to her life and extending her family. For this second edition of the book, we have made a few changes. We have included more information about evaluation roles and approaches. We have updated many of the sample forms based on additional experience and perspectives, and paired the samples with analysis and communication recommendations to make them easier to use and interpret. We have also included a broader array of examples in an attempt to expand the contexts discussed in the book. We have included more references to resources that we think are helpful for developing a deeper understanding or skill set that we were not able to fully address in this book or that may not be relevant for everyone.
Preface
fpref.indd xv
xv
2/1/08 4:02:36 PM
fpref.indd xvi
2/1/08 4:02:37 PM
I N T R O D U C T I O N
N
o matter how effectiveness is defined, there is a constant drive for efficiency, productivity, and stewardship of resources. This drive pushes commercial and not-for-profit organizations to weigh options, balance tradeoffs, and make better decisions about internal processes, customer or client relations, product releases and distribution, new programs and services, and other issues. An evaluation system provides a logical and practical framework for collecting, assimilating, and communicating information. The approach to evaluation we present in this book can be applied in a variety of contexts, but our focus is the evaluation of leadership development initiatives. Although many professionals do difficult and excellent work in and with organizations and communities to build leadership capacity, tools for evaluating leadership development are not as readily available. Our expertise in this area is coupled with a responsibility to share what we’ve learned. Our experience evaluating leadership development initiatives has given us specific ideas for how that work should be conceived and conducted: that it should be participatory, integrated with initiative design, and used to enhance learning and decision making. Using a participatory approach means involving key stakeholders—people who are, or will be, affected by the initiative being evaluated and/or by the outcomes of the evaluation. If lessons from an evaluation are to be successfully applied in an organization or community, all relevant stakeholders must be involved in the
xvii
flast.indd xvii
2/1/08 4:03:04 PM
planning or informed about the goals and objectives of both the initiative and the evaluation. As much as possible, they should also be involved in the interpretation of results and the creation of action plans based on the results because this ensures that the evaluation takes into account their perspectives and will secure their endorsement of the results and the plans that arise from them. Ideally, the discussions about and design of an evaluation are tightly integrated with the design of the initiative it evaluates. Both activities require a targeted discussion of the intended impact of the initiative. When collaborative processes are used to focus the evaluation and to apply the results, and when evaluation work is integrated into the design and implementation of an initiative, both the initiative and the evaluation are more effective, and learning can result. That learning is reflected in increased knowledge regarding barriers to and facilitators of change. In identifying how and where an initiative has been successful (or not), stakeholders can learn what processes, structures, areas, or systems support change as a result of the development initiatives, as well as what may need attention to be most effective. In addition, roles and responsibilities are further clarified during the evaluation planning process. That process also includes strategies for sharing results and lessons with a broad group of people. Evaluation helps a variety of people and groups of people meet several goals. For example, it helps them make informed decisions about how to improve development initiatives and examine the degree to which development goals have been accomplished and what work remains to be done. If development outcomes aren’t met, evaluation helps people understand why outcomes were not met, learn from their experiences, and apply those lessons to improving individual, team, organizational, and community effectiveness in terms that make sense for their context. These characteristics make an evaluation system not only important but also necessary for success.
xviii
flast.indd xviii
Evaluating the Impact of Leadership Development
2/1/08 4:03:04 PM
Understanding the Evaluation Function and Process
0
chapter ONE
T
here are many reasons to evaluate leadership development. If you are reading this book, you have probably already decided that you are interested in evaluating leadership development, and a significant portion of this book is designed to help uncover the purposes specific to your situation. We thought it would be helpful to share with you some of the common reasons people evaluate leadership development efforts (Hannum, Martineau, & Reinelt, 2007, p. 8): • To demonstrate more fully how participants, their organization, and communities do or might benefit from their leadership development program experiences • To fine-tune a proposed or existing leadership development intervention so that it has farther reach and might better meet its goals • To show how participation in leadership development experiences connects to such visions as improving organizational performance or changing society for the better • To promote use of learning-centered reflection as a central evaluation activity • To pinpoint what leadership competencies are most appropriate in particular settings • To encourage more comprehensive discussion about what works and why
1
c01.indd 1
2/1/08 3:51:20 PM
Evaluators come from a variety of educational and professional backgrounds. Research and evaluation are often seen as involving the same tasks and skill sets. However, while the two can be very similar, there are important distinctions. Evaluation traditionally focuses on determining the quality, effectiveness, or value of something, while research seeks to understand relationships among variables or describe phenomena on its way to developing knowledge that can be generalized and applied. When possible, it is good to seek someone who has extensive evaluation experience and training that is relevant to the context in which the leadership development program is operating. In general, evaluators can be classified as either internal evaluators or external evaluators. Internal evaluators are part of the program, organization, or community in which they conduct evaluations. External evaluators are hired from outside the program, organization, or community in which leadership development is taking place to conduct the evaluation. Internal evaluators usually have the benefit of a deep understanding of the context of the program and evaluation. They may also have developed credibility and trusting relationships enabling them to gather data that is more relevant and candid than data gathered by an outsider. However, because they are part of the organization or group they are evaluating, internal evaluators may be seen as biased and may also take aspects of the context or initiative for granted when an external evaluator would probe more deeply and challenge assumptions. External evaluators have the benefit of being able to ask questions that might seem naïve for an internal evaluator to ask. External evaluators may also seem more objective and less biased to stakeholders. If an evaluation requires asking sensitive questions, an external evaluator may be a better choice because individuals may be more willing to share that information confidentially with someone from outside. However, external evaluators may not fully understand the context of an initiative and evaluation, and may therefore be less sensitive to potentially offensive language or make recommendations or choose approaches that are less relevant. If at all possible, a combination of internal and external evaluators brings the best of both roles to the evaluation—their strengths complement each other. If funding is not sufficient for both and only internal evaluators are used, another strategy is to use an external evaluator as an advisor for certain parts of the evaluation. Doing so will cost less than using an external evaluator to conduct the evaluation, but it will provide some of the credibility that is sometimes more
2
c01.indd 2
Evaluating the Impact of Leadership Development
2/1/08 3:51:20 PM
likely to be associated with external perspectives. Note that we are not suggesting that internal evaluators lack credibility—just that some organizations place a value on external perspectives that should be considered. If you are pursuing an internal evaluation approach, but do not have individuals trained in evaluation, consider investing in training for those selected or volunteering to conduct evaluation. There are many colleges and universities that offer courses in evaluation as well as evaluation groups that offer workshops. If money is not available for courses or workshops, then you may want to build a small library, or at least purchase a book, to help build evaluation capacity. Hallie Preskill and Darlene Russ-Eft’s book Building Evaluation Capacity: 72 Activities for Teaching and Training (2004) is a practical choice. Another way to build capacity is to start an evaluation learning group. The group could meet to discuss shared reading on evaluation and/or to help each other think through evaluation issues. If you’ve decided on using an external evaluator but aren’t sure where to locate one, the web page for the American Evaluation Association (www.eval.org) includes a search function to locate evaluators. Many, although not all, of the evaluators listed on the site are located in the United States. Other professional evaluation associations also have searchable lists of evaluators. A list of evaluation associations and their websites is available in the list of resources on pages 117–128. Once you have identified an evaluator or group of evaluators, an important consideration is being clear about role expectations. Evaluators can play many roles. This list below (Hannum, Martineau, & Reinelt, 2007, pp. 8–9) outlines some of the more common roles played by evaluators. These roles are not mutually exclusive; an evaluator can play a combination of roles, as well as play different roles over the course of an evaluation. It is important to understand what the evaluator and those working with the evaluator are expecting in terms of the role the evaluator will play. • Assessor. Evaluators assess the value and quality of a leadership development program or intervention in order to determine whether it has achieved its desired outcomes without doing harm or provided a valuable return on investment. • Planner and designer. Evaluators assist stakeholders in using evaluation findings and processes to improve and sometimes to design a new program or
Understanding the Evaluation Function and Process
c01.indd 3
3
2/1/08 3:51:21 PM
intervention. They also engage designers to identify what outcomes are desired, what will demonstrate success, and what program elements will contribute to or cause these outcomes. • Trainer and capacity builder. Evaluators educate stakeholders so that they might design, implement, and use evaluation effectively. Often this is done by facilitating gatherings in which stakeholders participate in the evaluation process and learn how to use evaluation tools. • Translator and boundary spanner. Evaluators cross boundaries to listen to and search for multiple perspectives and interpretations. As they move back and forth across boundaries, evaluators carry perspectives and findings with them and share those with the other groups in ways that those groups can hear and understand. • Advocate. Evaluators present evaluation findings in public forums that are intended to influence decisions about a program, policy direction, or allocation of resources. Evaluators can give voice to ideas, perspectives, and knowledge that normally go unheard or unknown because the groups that espouse them are ignored by groups with more resources and power. Evaluators advocate for taking the time and investing the resources to reflect, inquire, study, and assess programs and interventions because this process increases the likelihood of success and impact. In their role as advocates, evaluators may find that they are asked to modify or couch their findings in ways that will have positive results for a particular audience. Evaluators have an ethical obligation to do their best to maintain the integrity of the evaluation. • Reflective practitioner. Evaluators learn from their own thoughts, reactions, and experiences through a systematic process of interaction, inquiry, and reflection.
THE CYCLICAL NATURE OF THE EVALUATION PROCESS Developmental initiatives often link different kinds of learning opportunities and occur over a period of time. They also link back to the efforts of an organization, community, or larger group so that individual development is connected to larger goals in a cycle of assessment, practice, and learning. We argue that the results of such initiatives are best measured with an evaluation process that is itself
4
c01.indd 4
Evaluating the Impact of Leadership Development
2/1/08 3:51:21 PM
cyclical—not isolated in its methods or defined by discrete points in time. Recognizing the cyclical nature of evaluations allows those involved to use them as planning and learning tools that augment the individual and group impact of leadership development. Change is the norm, and evaluation can be a tool for enhancing and dispersing learning amid what can be seen as continuous transition. This approach creates a fluid process for evaluating leadership development initiatives while enhancing individual, group, organizational, community, or collective learning, rather than creating a measurement system designed solely to create valid results (Preskill & Torres, 1999). It requires groups to think about creating or modifying structures and systems in order to incorporate evaluation more deeply into the culture. CCL has based its framework on the approach described by Preskill and Torres (1999), with some modifications in terminology to reflect its focus on leadership development.
THE CONTEXT FOR EVALUATION Because people have different worldviews and value systems, appropriate data gathering, synthesis, and interpretation require more than applying a set of tools. To be relevant and useful, data collection, analysis, and dissemination strategies need to “take into account potential cultural and linguistic barriers; include a reexamination of established evaluation measures for cultural appropriateness; and/or incorporate creative strategies for ensuring culturally competent analysis and creative dissemination of findings to diverse audiences” (Inouye, Cao Yu, & Adefuin, 2005, p. 6). Practicing culturally competent evaluation involves understanding how history, culture, and place shape ways of knowing and the ways in which knowledge is used. If you are working in a culturally diverse context (and most of us are), it is important to examine and understand how cultural differences may influence or impact leadership development and the evaluation of leadership development (see, for example, Chapters 12 and 13 of The Handbook of Leadership Development Evaluation (Hannum, Martineau, & Reinelt, 2007), as well as publications available on the California Endowment website: www.calendow.org or www .calendow.org/evaluation/reports.stm (such as www.calendow.org/evaluation/pdf/ OverviewBook.pdf ).
Understanding the Evaluation Function and Process
c01.indd 5
5
2/1/08 3:51:22 PM
THE FRAMEWORK USED IN THIS BOOK The first phase of our framework, focusing the evaluation, includes learning and planning activities that guide evaluators and others toward results that are relevant and beneficial (see Chapter 2). Activities that focus an evaluation include the following: identifying stakeholders for the initiative and for the evaluation, determining the purpose of the initiative and the evaluation, identifying the resources available, determining the level and type of impact, surfacing expectations, and drafting evaluation questions and potential data-collection methods. Ideally, evaluators conduct these activities in conjunction with the design or implementation of the initiative. The complexity of contexts and cultures combined with the complexity of developing leaders requires processes to help stakeholders develop a common understanding of issues, purposes, and roles. Combining the design phases of the evaluation and the initiative helps ensure the utility and efficiency of both processes. As you focus your evaluation, this combination will help you ask the appropriate people the right questions. At the end of this phase, you will have developed your evaluation strategy. Once the focusing activities are complete, the next phase is designing and conducting the evaluation, covered in Chapter 3. In this phase, evaluators and key stakeholders design and apply the evaluation plan. Measuring and interpreting degrees of change are complex endeavors. It’s at this stage that you will address research design considerations such as using multiple data-collection techniques. To manage the evaluation project’s complexity, you can use the information stakeholders have provided during the focusing stage to explore the benefits, drawbacks, and caveats associated with different evaluation techniques. The final phase in the cycle, communicating and using evaluation findings (see Chapter 4), is often overlooked. Typically, the results of the evaluation are compiled and reported by the evaluation team, which includes evaluators and key stakeholders (supporters, staff, and participants, for example), and this team may also make preliminary recommendations for action. Our position is that learning and action are the ultimate goals of a well-conceived and professionally produced evaluation. Delivering a report with recommended actions isn’t enough to realize that goal. Implementing and monitoring the action plan that arises from your evaluation help ensure that such learning will take place.
6
c01.indd 6
Evaluating the Impact of Leadership Development
2/1/08 3:51:22 PM
Focusing the Evaluation
0
chapter TWO
T
he ideal time to plan an evaluation is when the development initiative is being designed. When planning is integrated, evaluation questions can be used to help design the development initiative so that it will be more likely to promote the desired results. This is an effective way to focus your evaluation on stakeholder expectations. However, it’s not always possible to design an evaluation during the design of the initiative. Often those funding or conducting leadership development don’t realize the need for or don’t want to focus on an evaluation until after conducting a development initiative.
In designing a focused evaluation, it’s also important to understand the context in which the leadership development and evaluation are taking place. For example, in an organizational setting an evaluator must consider the ebb and flow of work and take critical times in an organization’s calendar into account. Many organizations have periods in their annual cycle that are particularly busy and stressful— bad times for evaluation activities. For example, if the people you wish to collect survey data from are enmeshed in annual budget planning, they may not have time to complete your survey. Every situation is going to be somewhat different. Making the evaluation process as simple and convenient as possible can ameliorate scheduling problems and help you gather the information you need. Whether you design your evaluation in unison with the creation of the development initiative or you design it after participants complete the process, it’s essential to define the key elements of your evaluation design. The following
7
c02.indd 7
2/1/08 3:51:56 PM
actions can help you successfully gather and focus the information you will need to design an effective evaluation. • Identify stakeholders. Who will be affected by the development initiative and its evaluation? • Define purpose. What are the reasons behind the development initiative? • Determine resources. Are there sufficient resources—money, time, staff, and evaluation tests and mechanisms—to support the evaluation? • Establish the desired types of impact. What are the expected outcomes? Is the development initiative expected to have an impact on individuals, groups, the organization, or the community? • Establish the period of time over which the impact is expected to occur. Is the development initiative expected to have short-term, midrange, or long-term impact? • Surface expectations. What is anticipated to occur as a result of the development initiative and its evaluation? What connections are important to the success of the initiative? What assumptions do different groups hold? • Determine and prioritize evaluation questions. How can stakeholder expectations be reflected in the evaluation questions being used?
IDENTIFY AND ENGAGE STAKEHOLDERS Stakeholders are people who are, or will be, affected by the initiative being evaluated and/or by the outcomes of the evaluation. If their questions are not answered, it’s unlikely that your evaluation will serve its purpose, no matter how rigorous your design. To avoid this disappointment and waste of resources and to get the highest-quality, most relevant data possible, it’s important to identify and include stakeholders early in the evaluation design process. Make all of your stakeholders aware that evaluation usually creates a demand for change—in individuals, in organizations, in communities, and in the initiatives on which the evaluation is focused. It’s important to have widespread agreement that stakeholder groups will use the findings of the evaluation to implement constructive change. It is also important to work with stakeholder groups to clarify the purposes and needs for evaluation (which we address in the next section). Use Exhibit 2.1 to identify the key stakeholders in the design and evaluation of the development initiative.
8
c02.indd 8
Evaluating the Impact of Leadership Development
2/1/08 3:51:57 PM
Exhibit 2.1. Stakeholder Identification List and describe the stakeholders who are involved in the development initiative or have an interest in the outcome of the initiative or its evaluation. In identifying stakeholder position or description, include such people as the designers of the initiative, trainers, human resources staff, potential participants, community leaders, senior managers or political leaders concerned with the results, managers whose staff will participate in the initiative, and the group or groups funding the initiative. In identifying stakeholder interest, make notes about the particular information needs your stakeholders have with regard to this initiative and evaluation. This list of questions isn’t exhaustive but offers guidance. If your circumstances suggest other questions, substitute them for these, or add them to this list. Who has an interest in the development initiative? Name: Stakeholder position/description: Stakeholder interest: Who has an interest in the evaluation’s processes and/or results? Name: Stakeholder position/description: Stakeholder interest: Are there additional people whose support (financial, political, organizational, etc.) is required for the success of the initiative or the evaluation? Name: Stakeholder position/description: Stakeholder interest: (Continued)
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
c02.indd 9
9
2/1/08 3:51:57 PM
Who has decision-making authority with respect to the initiative and/or the evaluation? Name: Stakeholder position/description: Stakeholder interest:
DEFINE PURPOSE Effective leadership development initiatives often link several different kinds of learning opportunities and occur over time, rather than as a single event. They often link back to group, organization, or community goals so that individual development is connected to broader goals. Typical learning opportunities in such initiatives might include an assessment-for-development initiative, one-onone coaching, skills-based training, challenging assignments, and action-learning projects (see Exhibit 2.2 for definitions of these components). The purpose of a development initiative may seem to be clear, but it’s important as you focus your evaluation to confirm that stakeholders have a shared understanding of that purpose. Designing the development initiative and its evaluation at the same time is an effective way to ensure that stakeholders have that understanding. Defining how your evaluation will measure the impact of the development initiative puts abstract goals (“We will develop better leaders”) into practical terms (such as “Our senior managers will understand how to give feedback to their direct reports” or “Our volunteer leaders will have more clarity about our social change vision and their part in the process”). Using multiple perspectives to confirm stakeholder assumptions will help you better define the purpose of the initiative so you can design a more effective evaluation. Although not all stakeholders will participate in determining the scope or focus of the evaluation, it’s important that you understand all of the assumptions the stakeholder group holds about the purposes of the development initiative. A thorough understanding of how elements of the development initiative fit together and the context in which the initiative and evaluation will take place are crucial to focusing your evaluation. A list of possible outcomes for leadership development can be found at www.leadershiplearning.org/system/files/Evaluating+Outcomes+and+ Impacts_0.pdf.
10
c02.indd 10
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:51:58 PM
Exhibit 2.2. Learning Opportunities Action-learning leadership project—a collaborative inquiry process in which participants work and reflect on real problems with learning partners, producing a tangible outcome while at the same time learning from the experience. Assessment-for-development initiative—an initiative in which data from assessments (usually including assessment instruments) are used to help participants identify their strengths and determine where they need further development. The assessment information in these initiatives is almost always used exclusively for development purposes and not administrative purposes (such as promotion or salary decisions). This type of initiative is intended to help participants assess current skills and behaviors, increase self-awareness, change perspective, and clarify values and goals. Challenging assignments—assignments that provide opportunities for individuals to learn and/or apply skills in a manner that would not be possible as part of their normal work responsibilities. Follow-through activities—designed to transition participants from a formal learning experience such as a program back to their work environment, carrying their leadership development goals back and gaining ongoing support for them. One-on-one coaching—a formal developmental relationship through which the participant engages in a series of one-on-one feedback sessions with a coach (the coach can be a person from his or her organization or someone outside the organization). Coaches can play a variety of roles, such as feedback provider, sounding board, feedback interpreter, challenger, and others. Skills-based training—a development experience in which individuals gain knowledge and practice behaviors necessary to hone present skills or develop new ones. The purpose of this type of training is to improve performance in a specified skill area.
Focusing the Evaluation
c02.indd 11
11
2/1/08 3:51:58 PM
In reviewing the leadership development initiative, evaluators can gain a full picture of the process by investigating several different avenues. Exhibit 2.3 can be helpful in directing your review.
Exhibit 2.3. Purpose Definition Seek answers to these questions from the initiative’s key stakeholders before designing your evaluation plan. The answers will help you define what stakeholders see as the purpose of the development initiative. With that information you can focus your evaluation to measure expected results. What specific challenge is the initiative expected to address?
How does this initiative support the organization’s business strategy or the community’s goals or social change strategy?
What specific leadership needs does this initiative address?
What are the purposes of the leadership development intervention(s)? For example, are stakeholders generally content with the status quo of leadership but seeking a standard of leadership practice? Do stakeholders want a program of development in order to create and reinforce a new and different set of skills?
(Continued)
12
c02.indd 12
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:51:58 PM
Are there any other external and internal pressures or demands for creating this initiative?
Will participants be held accountable for their development as a result of this initiative? If so, how?
What level of accomplishment is the initiative intended to promote? Knowledge acquisition? Awareness change? Behavioral change? Skill development? Performance improvement?
What type of impact is the initiative expected to have? Will it affect only individuals? Will it affect teams or groups? Will it have broad organizational or community impact?
How will the information in the initiative be delivered and over what span of time? Will it be a single five-day session or two three-day sessions held six months apart? Will it include online components?
(Continued)
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
c02.indd 13
13
2/1/08 3:51:59 PM
What data will be collected during the initiative that may be useful in an evaluation? What data are being collected by others that may be useful in an evaluation?
What evaluation techniques, such as end-of-program surveys, are already designed for or in use by the initiative?
What assessment instruments will the initiative use that could also be used to measure change during the evaluation?
What kind of information about the impact of the initiative do various stakeholders need? What will they do with the information? Why do they need it?
Logic models, theories of change, and pathway maps are other ways to map an initiative and an evaluation (see, for example, Chapter 2 of The Handbook of Leadership Development Evaluation [Hannum, Martineau, & Reinelt, 2007]). Asking key stakeholders to participate in and/or review logic models or pathway maps will help ensure that the purpose and process are clear. You can learn more about logic modeling on the W. K. Kellogg Foundation website: www.wkkf.org/ Pubs/Tools/Evaluation/Pub3669.pdf. In some cases, developmental evaluation may be more appropriate. This approach, described by Patton (1994), uses
14
c02.indd 14
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:51:59 PM
evaluation to empower program participants and staff, and is similar to the empowerment evaluation approach (most commonly associated with Fetterman and Wandersman, 2004). The role of the evaluator changes from judge to facilitator. Program participants and staff take time to develop their own criteria for what constitutes a good evaluation. Whatever approach you choose, make sure the purpose of the initiative and the evaluation are clear to the key stakeholders.
DETERMINE RESOURCES To design an effective, focused evaluation, you will need to know what resources (money, time, staff, and evaluation tests and mechanisms) are available. Evaluations typically take up 5 percent to 20 percent of the cost of a development initiative. As a general rule, the cost of the evaluation depends on the complexity of the initiative and the evaluation. The following are important questions to ask: When are stakeholders expecting to see results? What skilled staff members are available to perform the evaluation? How much data will be collected during the evaluation and how? How will evaluation results be used? You will need to make resources available to collect the necessary data, analyze it, and communicate the results. If stakeholders want to measure results at multiple stages, you will have to allocate your resources appropriately to meet that request. Skill sets you will want to tap for conducting an evaluation include database experience, statistical knowledge, survey development, interviewing experience, and project management. If there are available staff members with these skills, that can reduce costs. Many evaluation approaches encourage stakeholders to be part of the entire evaluation process, not only as advisors but as gatherers and analyzers of information. This approach is often referred to as participatory evaluation and it can reduce costs. Finally, using existing survey platforms, data analysis software, and other tools can also reduce costs. TYPES AND AMOUNT OF IMPACT Organizations and foundations typically evaluate their systems and processes to determine whether expectations have been met. When it comes to leadership development initiatives, organizations and foundations frequently expect a demonstrable return on their investment. Stakeholders assume that development
Focusing the Evaluation
c02.indd 15
15
2/1/08 3:51:59 PM
initiatives will affect the individuals who participate, the groups or teams to which they belong, and the organization or community as a whole. But they may not expect impact at all levels (individual, group, team, organization, and/or community) for every initiative. Understanding the type of impact the initiative is intended to have and the type of impact stakeholders expect it to have can strengthen your evaluation design by clarifying the goals you will measure against. As you consider the type of impact, also think about the degree to which each type of impact is expected or desired. Is it enough that things get better generally, or is a certain amount of improvement or change desired? Your evaluation design is stronger and more apt to provide relevant results when it focuses on stated goals and expectations. According to Figure 2.1, this could mean that • Individuals will be better able to perform their leadership roles; • Groups may be better able to perform effectively; • An organization may experience an improvement in climate and/or in the bottom line; • A community group may see improved effectiveness of community centers, collaboration across community resources, and improved outcomes in a specific target of the community;
Figure 2.1. Expectations for Leadership Development
Improvement in skills or changes in behaviors of individual leaders
Improvement in the group or team effectiveness
Enhanced ability to meet organizational, community, system, government, or other broad goals
Adapted from McCauley, Moxley, and Van Velsor, 1998.
16
c02.indd 16
Evaluating the Impact of Leadership Development
2/1/08 3:52:00 PM
• A foundation may see improved ability to implement effective, high-impact programs; • Systems may see improved coordination between segments of the system; and • Governments may see improved services and use of resources. Individuals A leadership development initiative should have an impact on the individuals who experience it. The impact itself, however, can vary. For example, individuals may learn new leadership models and practices, or they may learn new practices specific to a particular business, community, or culture. Participants may develop an increased awareness of their personal leadership styles and how they affect others. They might change work-related behaviors or increase their effectiveness using newly acquired skills. Impact will vary, depending on the content and design of the initiative and the development needs of individual participants. For example, threeto five-day assessment-for-development initiatives are likely to result in participants’ acquiring critical knowledge, building awareness, and gaining some ability to apply lessons to certain behaviors and situations. For changes to become ingrained in an individual’s performance, additional developmental experiences, such as oneon-one coaching and challenging assignments, are often necessary. Challenging assignments provide the opportunity for individuals to learn and apply skills in a manner that is not possible as part of their normal work responsibilities. Teams/Groups Development initiatives created for individual leaders can also have an impact at the team or group level. For example, a leader’s group might perform more effectively after that leader has enhanced his or her leadership capability. The group may be able to get products to market more quickly because its manager has developed skill in focusing group effort. After participating in a development initiative and learning skills that encourage better communication among team members, a team leader may be able to lead the team toward more effectively supporting community goals. When development initiatives are targeted at intact teams, outcomes are more pronounced and more quickly observable because the team as a whole is able to put the members’ new awareness, knowledge, and skills into practice immediately.
Focusing the Evaluation
c02.indd 17
17
2/1/08 3:52:00 PM
Organizations When organizations invest substantial resources in developing leadership capacity, they often look for results that will help them achieve strategic objectives, sustain their effectiveness, and maintain their competitive position. The quality of an organization’s leadership is only one contributing factor to reaching those goals. Even so, a leadership development initiative can, among other things, facilitate culture change, enhance the organizational climate, improve the company’s bottom line, and build a stronger, more influential organizational profile when all leaders in the organization have participated in it. An organization may also use development initiatives to augment the internal branding of the company name, as is the case when a company offers development opportunities to present itself as an appealing place to work. With regard to evaluation, stakeholders will want to gather information that provides evidence of the link between leadership development and appropriate measures of organizational success. Communities Leadership development programs may seek changes in geographic communities or communities of practice (Hannum, Martineau, & Reinelt, 2007). Evaluators look for changes in the composition of leaders who are in decision-making positions, in social networks, in partnerships and alliances among organizations, in ways in which emerging leaders are identified and supported, and in the numbers and quality of opportunities for collective learning and reflection. Fields Leadership development programs may seek changes in language, paradigms, and how knowledge is organized and disseminated (Hannum, Martineau, & Reinelt, 2007). Evaluators look for changes in language, shifts in paradigm, the demographics of participants in a field, and the visibility of ideas within a field. Networks In a community or field context, when network building is a core focus of the leadership development effort, evaluators may look for changes in the diversity and composition of networks, levels of trust and connectedness, and their capacity for collective action (Hannum, Martineau, & Reinelt, 2007).
18
c02.indd 18
Evaluating the Impact of Leadership Development
2/1/08 3:52:01 PM
Societies/Social Systems Leadership development evaluators sometimes seek to measure or capture social or systems change (Hannum, Martineau, & Reinelt, 2007). Because this type of change typically takes longer to occur, it may be difficult to see in the timeframe of most evaluations. Evaluators look for changes in social norms, social networks, policies, the allocation of resources, and quality of life indicators.
IMPACT OVER TIME If the impact of an initiative is expected to occur over a period of time, you can design your measurements to account for short-term, midrange, and long-term impact. Design your evaluation plan and choose your measurement and communication plan to address each of these timeframes. The short-term impact of a development initiative can include what participants think about the initiative and their experience with it immediately after completion. Short-term impact may also include the development of new ideas or new self-awareness based on what participants have recently learned from their developmental experience. Midrange impact usually occurs three to six months after the development initiative ends. Measurements at this time usually relate to individual skill improvement, behavioral change, or team development. Assessing a development initiative’s long-term impact occurs nine months to a year (or more) after the initiative ends. Areas that benefit most from this type of evaluation include performance improvement, the attainment of more complex skills, and organization- or community-level change. Measuring change over time allows you to see the trends in changes to determine when change occurs and whether change is enduring. SURFACE EXPECTATIONS An organization or foundation often conducts evaluations to determine whether its expectations have been met. Sometimes there is an expectation that there will be a demonstrable return on investment. But more often there are expectations that may not be defined or explicit. There may be multiple reasons that stakeholders see for investing in leadership development. Different stakeholders may have different, possibly conflicting, expectations. Failing to understand and address those expectations can have negative consequences for the initiative and its evaluation.
Focusing the Evaluation
c02.indd 19
19
2/1/08 3:52:01 PM
Some common expectations for leadership development initiatives are that participants will become better leaders and will share a common language of leadership (for example, they will learn and be able to put into practice specific leadership terms, models, and styles) so that they will understand how to work with each other more effectively. Organizations may also expect it to be easier to find and retain talented employees because developmental opportunities exist. Other expectations include making the organization a more engaging place to work, delivering products or services to market and to clients more quickly, and increasing revenue in the case of for-profit companies or broadening and deepening community impact in the case of not-for-profit organizations. Many leadership development initiatives seek to create connections between leaders within or between groups. Using logic modeling or pathway mapping (see, for example, Chapter 2 of The Handbook of Leadership Development Evaluation [Hannum, Martineau, & Reinelt, 2007], as well as www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf ) provides a tangible way to facilitate and document stakeholders’ expectations and understandings of an initiative. The EVALULead model is another approach that may prove helpful (see, for example, Chapter 3 of The Handbook of Leadership Development Evaluation, as well as www.leadershiplearning.org/community/files/download?version_id=2127). A development initiative designed to meet objectives is linked closely to stakeholder expectations. Integrating the design of your evaluation with the design of the initiative will help you measure the right results. Part of that integration process is linking your evaluation plan to stakeholder expectations. Use Exhibit 2.4 to specify implicit and explicit expectations so that you can take them into account during the design phase. Note that the questions in this exercise are intentionally overlapping, framed in different ways to elicit comprehensive ideas.
20
c02.indd 20
Evaluating the Impact of Leadership Development
2/1/08 3:52:02 PM
Exhibit 2.4. Surface Expectations Use this worksheet to specify implicit and explicit expectations so that you can take them into account during the design phase. If you work with different stakeholder groups separately, you may need to change the focus based on what is relevant for each person or group. If it is possible, it can be helpful for the different stakeholder groups to get together at the same time to discuss expectations. What specific outcomes have stakeholders said they expect in order to consider the initiative a success?
Identify specific behaviors that stakeholders expect participants to exhibit as a result of this initiative.
How are these behaviors different from or similar to current behaviors being exhibited?
Is there baseline information about participants’ current behavior? (Review previous assessment activities, if available.)
(Continued)
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
c02.indd 21
21
2/1/08 3:52:02 PM
What are the implications of not pursuing a development initiative or an evaluation?
Over what period of time is the initiative to occur?
When do stakeholders expect the initiative to have its desired impact? (How much time will they allow before they expect to see the desired change?)
What questions will stakeholders and others ask about the effectiveness of the evaluation?
What evidence of impact will the stakeholders consider necessary to believe that the intended outcomes have been achieved? What type of data will they accept (qualitative or quantitative)? What sources will they trust (for example, participants, their managers, their clients)?
How do stakeholders expect results to be communicated (final evaluation report, update memos, etc.)?
22
c02.indd 22
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:52:03 PM
DETERMINE AND PRIORITIZE EVALUATION QUESTIONS Once you understand the expectations of all key stakeholders for both the development initiative and its evaluation, you can define the questions the evaluation should answer. It’s important to understand the difference between evaluation questions, which define a broad but intentional direction, and survey questions, which are created specifically to generate data for analysis. Evaluation questions should be well defined and linked specifically to clear objectives. The questions should be informed by stakeholder expectations so that they appropriately address those specific concerns. If the questions are not linked to stakeholder expectations or are otherwise unclear, you may come up with data that you can’t interpret or communicate with respect to the goals set out in the development initiative. Evaluations are frequently designed to answer multiple questions. However, we recommend that your evaluation address only a few key questions to keep the evaluation goals clear and to maintain a focused effort during the implementation stage. The questions in Exhibit 2.5 will help you determine and define questions to use in your evaluation.
Focusing the Evaluation
c02.indd 23
23
2/1/08 3:52:03 PM
Exhibit 2.5. Developing Evaluation Questions Use this worksheet to determine and define questions to use in your evaluation. What are the critical questions the evaluation should answer?
From whose perspective are the questions being posed?
When are answers to those questions expected?
What are the objectives of the development initiative?
What aspects of the initiative address those objectives?
What logical connections can be made (or should be investigated) between initiative outcomes and the intended impact?
(Continued)
24
c02.indd 24
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:52:03 PM
What types of impact and what timeframe are of interest to stakeholders?
What outcomes are possible to measure, given the timing of the evaluation in relation to the implementation of the initiative?
What elements of context are important to understand?
Who will use information from the evaluation, and how will information be used?
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
c02.indd 25
25
2/1/08 3:52:03 PM
Developing questions using the aforementioned worksheet as a guide, several specific questions related to a particular development initiative can be investigated. Those questions might include the following: • To what extent does the leadership development initiative meet its stated objectives? • Are there any unintended benefits or challenges raised by the initiative? • To what degree are participants prepared to apply what they have learned to their work? • To what degree have participants applied what they have learned to their work? • To what extent have participants made significant behavioral changes? • What is the impact of participants’ behavioral changes (or other changes) on those around them? • How has the organization or community experienced the intended changes (benefits) as a result of the initiative? Target Population The outcomes of a leadership development initiative depend heavily on the individuals involved. The skills and perspectives that they bring to the initiative and the context in which they work affect what they are able to learn and the results they are able to achieve. Therefore, it’s critically important for you to fully understand the target group for the development initiative you are evaluating. Exhibit 2.6 can be helpful.
26
c02.indd 26
Evaluating the Impact of Leadership Development
2/1/08 3:52:04 PM
Exhibit 2.6. Target Population It’s critically important to fully understand the target group for the development initiative you’re evaluating. Use this worksheet to determine the target population, identify what those individuals bring to the development process, and specify how best to measure the results of the process. Who is the target population for this initiative (shop floor supervisors, managers of volunteers at a social services agency, organizational executives, etc.)?
Why do these individuals need a development initiative? Why does this initiative focus on them in particular?
What type of training or development has this group experienced in the past? What has their reaction been? What has been the impact of prior training and development?
How do the participants view this particular initiative? What positive and negative associations does it have from their perspective?
(Continued)
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
c02.indd 27
27
2/1/08 3:52:04 PM
Will participants be held accountable for their development? If so, how and by whom?
What development barriers and facilitators will participants likely encounter? For example, if a new reward system is being implemented at the same time participants are expected to make significant behavioral changes, will the reward system help them or prevent them from making changes?
Is the initiative you are evaluating part of a larger leadership development process? If so, how are they integrated?
28
c02.indd 28
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:52:04 PM
Managers of and Those Working with the Target Population Because the success of a development initiative is affected by its context, evaluators may have to become familiar with the managers to whom participants report and those with whom participants work. The support and involvement of others have an impact on participants’ ability to effectively integrate what they’ve learned and apply the skills they’ve acquired. In particular, managers of participants are an important group to consider. Exhibit 2.7 can be helpful in defining the managers who are likely to be most affected by the development initiative and what role they might play in the initiative.
Focusing the Evaluation
c02.indd 29
29
2/1/08 3:52:05 PM
Exhibit 2.7. Managers of and Those Working with the Target Population Use this worksheet to define people, other than the participants, who are likely to be most affected by the development initiative and what role they might play in the initiative. To whom do the participants report? Who are their managers?
With whom do participants work?
To what extent do managers of participants and others have to be oriented to the development process?
What content for and approach to orientation would be important? For example, will the organization provide a letter explaining the process, defining expectations, and outlining the type of support that’s required and expected?
Will participants’ managers or others be held accountable for participants’ ongoing development? If so, how? Must that information be tracked? How might it best be tracked?
How might the managers or others hinder participants’ development efforts or the outcomes of the initiative?
30
c02.indd 30
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:52:05 PM
Prioritizing Evaluation Questions Prioritizing evaluation questions serves two functions. The first is to reduce the overall number of questions. If a question is not important, should resources be expended to address it? The second function is to determine the relative merit of the different questions. Some questions are bound to be more important than others, and it’s helpful to know what is critical versus what is just interesting. There are multiple ways to prioritize evaluation questions, from having stakeholder groups vote or rate the importance to asking funders to decide what they are willing to provide resources for.
IDENTIFY THE SAMPLE AND CHOOSE A DATA - COLLECTION METHOD As you prepare to match your evaluation questions with data-collection strategies, consider which groups of people are likely to have the information you need to answer your questions. Reviewing your list of stakeholders can help you target the appropriate people for data collection. You may want to collect data from an entire group, or you may want to collect data from a sample. For example, information about the trainers’ behavior in the classroom may best be collected from participants, because other stakeholder groups are unlikely to have direct information about that experience. Conversely, participants may not have detailed information about such matters as the relationship of organizational HR policies to leadership development success. The complexity of your evaluation questions might also influence your choice of sample respondents. For example, are there multiple perspectives that you need to capture from a diverse group of stakeholders? Another question to ask when identifying your sample is: What group will be able to provide credible information and maximize the use of the evaluation results? For example, if your evaluation questions are designed primarily to inform a particular group and you know that group does not find the XYZ team a credible source of information, it will be important for you to obtain information from another team instead of or in addition to the XYZ team. It’s difficult to achieve this level of knowledge about political underpinnings, so you may have to invest time in earning the trust of stakeholder groups. When it comes to selecting a data-collection method, keep in mind that no single method (survey, interview, observation, or assessment instrument, for
Focusing the Evaluation
c02.indd 31
31
2/1/08 3:52:05 PM
example) can fully tap the opinions and experiences of respondents. Therefore, the most effective evaluations use more than one method. This allows the strengths of one method to compensate for the weaknesses of another—what’s called data triangulation. The more important, complex, and/or potentially controversial an aspect of the evaluation is, the more data and types of data may be needed to respond to stakeholder expectations. Depending on your audience and the type of information you need, either qualitative or quantitative data may be more appropriate. In many situations, combining qualitative and quantitative data can provide powerful evidence for understanding and demonstrating impact. You must also consider the resources available to you for the data-collection and analysis stages of your evaluation. The following questions illuminate inherent strengths of different data-collection approaches. During the design of your evaluation, use them to guide your thinking about the possible approaches you might take. Select those collection methods that have the best chance of obtaining the information you require to carry out an effective evaluation. (More detailed information about specific data-collection techniques can be found in Chapter 3.) Which Methods Are Best Suited for Answering Your Evaluation Questions? If your question asks how often or how much, a quantitative approach (such as a survey or an assessment instrument that gathers frequencies or percentages) would be appropriate. But if your question asks in what ways or how, a qualitative approach (such as face-to-face interviews or focus groups) might be a more effective collection method. What Is the Purpose of Your Evaluation? How Will the Information Be Used? Evaluations typically seek to provide evidence or information related to one or more areas (refer to the list at the beginning of Chapter 1). The answers to questions defining purpose and use provide insights into the kinds of information required. Once you have a clear purpose in mind, consider who will use the information you intend to gather and how. For example, while stories and survey results can provide information about the same thing, one or the other may provide information
32
c02.indd 32
Evaluating the Impact of Leadership Development
2/1/08 3:52:06 PM
that is a better fit for the purpose or audience of the evaluation. Stories may provide rich context and meaning, while aggregated survey data give a sense of the pervasiveness of impact. Think about the stakeholder groups with whom you will share the data (for example, funders, program staff, community members, executive teams). What kind of evidence will be credible or meaningful to them? Knowing your audience may help you determine that one method is more appropriate than another. Because different kinds of data lead to different kinds of understanding, it’s important for you to think about how useful certain kinds of data will be as stakeholders look toward putting results into action. But be careful not to allow the preference of any stakeholder group to determine completely what data you collect and how. While stakeholder preferences are important, the methods you choose should always be appropriate for the evaluation question you seek to answer. What Motivated the Need to Evaluate in the First Place? If the goal of your evaluation is to gather information from a range of individuals, a quantitative method (such as a survey) might be the most efficient and costeffective method. But if the purpose is to understand the complexities of a situation (to gain more in-depth information), qualitative methods such as face-to-face interviews or focus groups are more effective. Your evaluation may also have both types of goals and therefore require both qualitative and quantitative methods. How Practical and Feasible Is Each Method? If you must conduct your evaluation quickly and assess a large number of people using limited resources, it might be best to use a quantitative method such as a survey or assessment instrument. If you have enough time and resources (both technological and human), you can reasonably choose from qualitative methods such as observation, telephone and face-to-face interviews, and focus groups. In addition to financial resources and time constraints, give careful consideration to the practicality and feasibility of using different approaches given the evaluation expertise available to you (data collection, management, analysis, and interpretation). For instance, if you plan to use interviewing as one of your data-collection techniques, how will you analyze that information? If you plan to collect survey data, how will you present the results? Creating an evaluation plan that links your evaluation questions with your data-collection techniques (see the evaluation plan design examples in Chapter 3)
Focusing the Evaluation
c02.indd 33
33
2/1/08 3:52:06 PM
is a useful strategy for developing a comprehensive and well-documented evaluation plan. Although it takes additional effort at the beginning of the evaluation project, it typically saves time (as well as confusion and disappointment) in the long run.
A PLAN TO USE EVALUATION RESULTS Development programs are not conducted without some expectation that participants will use what they’ve learned. Likewise, your evaluation won’t be worth much if its results are not used. Evaluators must work together with key stakeholders to achieve the most effective use of the results. Understanding and making explicit the purposes and benefits of evaluation can facilitate that process. Information from evaluations can, and should, be used to enhance learning. You can help organizations, communities, and other groups with this task by planning how you will communicate the results of the evaluation and how the organization and/or community can use those results in strategic ways. (Read more about the use of evaluation results in Chapter 4.) As you focus your evaluation, consider how you can measure and communicate information about a development initiative’s impact so that those who can and want to use the results can do so in a meaningful way—to gain deeper knowledge of process and systems, to pass on to others the lessons participants have learned, and to design more effective development opportunities.
34
c02.indd 34
Evaluating the Impact of Leadership Development
2/1/08 3:52:07 PM
Exhibit 2.8. Focusing the Evaluation Checklist ❑ Plan your evaluation when the development initiative is being designed.
❑ Identify stakeholders with an interest in the initiative and its evaluation.
❑ Define the purpose of the evaluation. ❑ Determine what resources are available for conducting the evaluation.
❑ Establish the types of impact the development initiative is expected to have.
❑ Establish the period of time over which the impact is expected to occur.
❑ Surface expectations about the development initiative and its evaluation.
❑ Determine and prioritize evaluation questions. ❑ Identify the development initiative’s target population. ❑ Identify the managers of and those working with the target population.
❑ Identify the sample and choose a method for collecting evaluation data.
❑ Make a plan for using the results of the evaluation.
Focusing the Evaluation
c02.indd 35
35
2/1/08 3:52:07 PM
Designing and Conducting the Evaluation
0
chapter THREE
A
fter you have focused the evaluation in its intent, you and the key stakeholders are ready to design it and carry it out. To do this, you will select the specific evaluation methods you will use, customize those methods to meet the specific demands of your evaluation plan, and then implement them. To help you make those selections and guide your implementation, the following evaluation design guidelines provide tactics and tips regarding the measurement of change. EVALUATION DESIGN GUIDELINES Follow these evaluation design guidelines whenever possible in designing your evaluation. They will help you improve the quality of the evidence you collect and present—making it more difficult for others to dismiss evaluation findings as an anomaly or the result of some measurement fluke or design flaw. Following these guidelines does not guarantee defensible results, but it does improve the likelihood that you can gather and present powerful evidence of impact. Additional guidance is provided by the Joint Committee on Standards for Educational Evaluation (1994). Examine Impact from Multiple Perspectives Different stakeholder groups have different perspectives on the impact of a development initiative. When evaluating skill levels at the start of the process and in evaluating how much an individual has changed at the end of the process, funders,
37
c03.indd 37
2/1/08 3:53:29 PM
board members, and program staff, for example, differ from each other and from participants themselves. Although for some purposes one perspective might be considered more relevant than others, evaluators should consider all perspectives, since knowing the whole picture is one of the most important elements in designing a valid evaluation. Consider who has the opportunity to directly observe or experience the type of changes you want to assess, and ensure that you collect appropriate data from those individuals. Assess the Different Kinds of Change That Can Be Observed Developing leadership capacity often involves multiple types of change (for example, new knowledge that brings enhanced self-awareness and prompts skill development), so understanding where an intervention may be having an impact and where it may be falling short of expectations is critical to designing an effective evaluation. Because individuals are ready for different kinds of learning or change at different times, it’s important that you design your evaluation to capture change as it occurs. Observable changes can include knowledge and awareness gains, individual behavioral change, individual or group performance improvement, and changes in the organization. Use Multiple Data-Collection Methods No data-collection method is perfect. Each has its benefits and shortcomings. An evaluation design that uses multiple methods can reveal information about all of the different kinds of impact stakeholders expect and can provide you with the most comprehensive sense of what kinds of change are occurring. For example, behavioral change might be best captured by a 360-degree instrument designed to measure change, but face-to-face interviews may be the best way to understand the obstacles people face in implementing desired behavioral changes. Endof-initiative evaluation surveys might be the best way to measure participants’ immediate satisfaction with a training initiative, but they don’t reveal issues of individual development in the way that questionnaires and interviews can. Look at Change Over Time Leadership development efforts often have short-term, midrange, and longterm effects. Limiting your evaluation to participants’ reactions immediately after an event, or even thirty days following the event, may seem like an efficient
38
c03.indd 38
Evaluating the Impact of Leadership Development
2/1/08 3:53:30 PM
way to quickly wrap up a study, but to accomplish significant change takes longer than that. There may actually be a period of reduced performance while participants are learning and practicing new skills, which is not necessarily indicative of a development initiative’s failure. If your evaluation looks only at the short-term results, it may not reveal significant long-term impact or may miss the opportunity to learn how to enhance long-term and more significant impact. Design your plan to measure change at multiple, and appropriate, points in time. Assess Change at Different Levels Although individuals are the participants in leadership development initiatives, organizations and funders usually expect that the development of significant numbers of individuals will have an impact at the group, organization, and/or community level and beyond. Restricting your assessment to individuallevel change (or group-level change, for that matter) doesn’t make clear the connection between the different levels of change. It’s important to measure both kinds of change and to analyze the connections between them to understand why a development initiative did or did not have the expected impact. Use Control Groups for Comparison Using control groups for comparison gives you greater certainty that the changes found in your evaluation are the result of the leadership development intervention and not of some other factor. Create your control group using people with characteristics (age, job type, and prior training, for example) that are similar to those of the people participating in the development initiative. If the intervention is the only known systematic difference between the two groups and the participant group shows more change than the control group (and nothing unusual occurred during the intervention and evaluation periods), you can reasonably argue that the change resulted from the initiative. More information about using control groups in leadership development evaluations can be found, for example, in Chapter 1 of The Handbook of Leadership Development Evaluation (Hannum, Martineau, & Reinelt, 2007). In many situations, a control group is not feasible or practical. In those cases, another option is a pilot study, shown in Exhibit 3.1. Exhibit 3.2 examines reliability and validity.
Designing and Conducting the Evaluation
c03.indd 39
39
2/1/08 3:53:30 PM
Exhibit 3.1. Pilot Study Is a Good Evaluation Tool A pilot study can act as a dress rehearsal for your evaluation. It can reveal unforeseen shortcomings in data-collection tools and help you hone your collection process and analysis strategies. The size of the pilot study depends on the method you’re using and the type of information you’re attempting to gather (Kraemer & Thiemann, 1987). If you are using assessment tools, investigate the quality of the assessments as part of the pilot (see Exhibit 3.2). Ideally, your pilot study would include an analysis of data to determine whether the measure is properly designed to deliver the desired kind of information. A pilot study has several benefits: you can determine whether responses seem appropriate, interview protocols can be tested to ensure that questions are clear and are eliciting intended responses, and surveys can be tested to see whether the questions are clearly written and the response options make sense. You may not have the time, resources, or opportunity to conduct a pilot test. Another way to ensure that you will collect good information is to form a data-collection review team. This team can review any surveys or protocols you develop for use in data collection and examine the process and analysis strategies you intend to employ. Members of the team might include content-matter experts (people who know about the subject area you are measuring), technical or measurement experts (people who know about measurement methods), population group experts (people familiar with the content and culture of the target population), and experts in the effects of bias, ethics, and similarly affective issues. Including stakeholders in the review process can be a way to build their engagement with and commitment to the evaluation work, while providing information that strengthens the evaluation.
40
c03.indd 40
Evaluating the Impact of Leadership Development
2/1/08 3:53:30 PM
Exhibit 3.2. Reliability and Validity In general, reliability is the consistency of an assessment, and validity is the accuracy of an assessment and the appropriateness of an assessment for a specific use. Neither reliability nor validity can be measured absolutely or established definitively for an assessment. Paying attention to the quality of assessment information and exploring what results mean and how they are used are important parts of the evaluation process. Reliability is the degree to which an assessment produces consistent results. If an assessment does not produce consistent scores, you may be receiving more error than information. Reliability is never truly measured, but it can be estimated. The same test will likely have different reliability estimates, depending on how reliability is calculated and the sample used. The appropriate reliability level depends on the situation. Reliability is usually reported on a scale ranging from 0 to 1, with estimates closer to 1 being preferred. There are three ways commonly used to assess reliability. Internal consistency provides information about whether questions on a scale are measuring the same concept. Interrater agreement provides information about the degree to which ratings from different sources agree. Testretest provides information about the stability of questions and scales over time. Validity is the combination of two ideas: the degree to which an assessment measures what it claims to measure and the usefulness and appropriateness of an assessment for a given purpose. Validity is an extremely important consideration when you are developing or using assessments. Validity is a multifaceted concept, and multiple types of evidence are needed to establish it. Evidence should be gathered in the varying situations and with the varying populations for which the assessment is intended. Validity has to do with the test, the people taking the test, the purpose of the test, and the consequences of the test. (Continued)
Designing and Conducting the Evaluation
c03.indd 41
41
2/1/08 3:53:31 PM
There are several types of validity evidence. Content validity is the extent to which assessment adequately and comprehensively measures what it claims to measure. Construct validity is carried in the relationship between test content and the construct it is intended to measure. Typically, this type of evidence involves logical and/or empirical analysis, including statistical comparisons to other assessments and expert judgments of the relationship between the assessment and the construct. Criterion validity is found in the relationship between the assessment and a criterion such as effective performance. Consider, for example, the connection between an assessment of job performance and job performance ratings. Concurrent evidence refers to evidence collected at the time the test is administered, and predictive evidence is evidence collected at a later time. We sometimes take for granted that an assessment is providing accurate, useful, and appropriate information. Assessments do not always do that. Validity studies are one way that question or test bias or unfairness can be revealed. Bias is the presence of a question or test characteristic that results in differential performance for individuals of the same ability but from different ethnic, gender, cultural, social status, or religious groups. Bias often stems from limitations of our perspective and understanding. No test is free from bias, but question and test bias and unfairness can be detected and reduced.
Use Time-Series Designs for Comparison It’s often difficult to put together a control group, so another way of determining the effect of a development initiative is to gather data from the participant group well before they participate in the initiative. Then you can look at trends of growth related to the period of time before the initiative and compare them with trends that occur during and after it. As long as no other changes that may have affected the outcomes have occurred during that same time period, you can infer any differences in trends to be the result of the development initiative. For more on time series designs, see Sawyer (1987) or the example on the Center for Problem-Oriented Policing website (http://popcenter.org/Tools/tool-assessing_appendixB.htm).
42
c03.indd 42
Evaluating the Impact of Leadership Development
2/1/08 3:53:31 PM
THE MEASUREMENT OF CHANGE: METHODS AND ISSUES While individuals can sometimes agree that a situation or an individual has or has not changed, they are much less likely to agree on how much change has occurred and whether the change represents an improvement. One primary cause of problems in measuring change is called response-shift bias. Responseshift bias occurs when individuals, having rated themselves at one time from one perspective (“I am a good leader”), change their responses at a later time, not because they see themselves as changed but because their perspectives have changed (see Exhibit 3.3).
Exhibit 3.3. Shifting Perspectives Make Measuring Individual Change Difficult Before participating in a leadership development initiative, managers rate themselves on a variety of questionnaires focused on leadership skills and perspectives. During the initiative, they are exposed to a variety of leadership models and to a variety of people whose perspectives about leadership differ from their own. As a result, they leave the initiative with a somewhat different idea of what effective leadership is. When they are asked after the initiative to rate their leadership skills and perspectives, it’s possible that they may rate themselves the same or even lower than they rated themselves before the initiative, even though they may have actually improved in some areas they have targeted for development. This is because they are using a different definition of effectiveness than they used before the initiative or they have higher expectations for their performance. The initiative may have had impact, but the measurement of its effectiveness (using a survey in this case) may not note any change. The same effect has been found in other rater groups, such as participants’ bosses, direct reports, and peers. This is especially true when survey questions are not specific enough to identify behaviors.
Designing and Conducting the Evaluation
c03.indd 43
43
2/1/08 3:53:31 PM
Evaluators who use multirater, or 360-degree, surveys to measure how a leadership development initiative has changed individual performance or behavior often use one of the following methods: pre- and post-initiative assessments, retrospective pretest and posttest assessments, and degree-of-change ratings. Each method has its own benefits and drawbacks. Pre- and Post-Initiative Assessments One of the most popular methods for measuring change is to use the same assessment survey before and after the leadership development initiative takes place. Some organizations may insist that a survey be readministered after an initiative has taken place because they find it too expensive to develop a new assessment survey. This technique works best in situations in which what you want to measure is very specific and concrete (easily observable). There are many factors other than the development initiative itself that could cause a change (positive or negative) in pre- and post-initiative assessments. Response-shift bias is one example; another is a change in the organization (restructuring, layoffs, or new compensation systems, to name a few). Retrospective Pretest and Posttest Assessments Retrospective pretest and posttest assessments require two ratings at the same time after participants have completed the initiative. One rating usually focuses on describing an individual participant before the intervention. The second rating assesses the person’s skills and behaviors at the time the survey is completed. The participants, their bosses, and other stakeholders can be involved in the rating process. It’s beneficial to include ratings from multiple perspectives. Some evaluators doubt the merits of this method, perceiving it to create a “demand characteristic” that would automatically result in increased ratings of effectiveness from the “before” to the “now” ratings. However, CCL’s research and other research in the field (Howard, Ralph, Gulanick, Maxwell, Nance, & Gerber, 1979; Pratt, McGuigan, & Katzev, 2000; Rohs, 1999; Rohs & Langone, 1997) argue for its validity. Ratings of change are highly correlated with objective measures of change such as performance appraisals. Degree-of-Change Ratings Another method for measuring change is to ask individuals to rate the degree of change using a response scale. For example, raters could select from a five-point
44
c03.indd 44
Evaluating the Impact of Leadership Development
2/1/08 3:53:32 PM
scale ranging from “no change” to “great positive change.” Research has shown this to be an effective method; there seems to be more agreement across rater groups (peers, direct reports, and bosses, for example) as to the amount of change when all groups are rating change directly, as compared with evaluations that measure change using pre- and posttest ratings (Peterson, 1993). Change can be either positive or negative. It is important to measure the direction of the change. 360-Degree Assessment Retests Many leadership development initiatives make use of 360-degree instruments, administered before the start of the initiative, as a way of providing participants with feedback regarding the state of their leadership capabilities. To measure change, some organizations like to use the same 360-degree instrument after the initiative. Using the same instrument lets organizations obtain another snapshot at a particular point in time from a particular group of people and lets them compare and contrast broad themes and patterns. Although this process seems to make sense, it does have some problems. For example, if different raters are used (which is often the case), there may be changes in scores simply because different raters are providing information. Also, participants revisiting the same 360-degree instrument will require access to someone familiar with both their original development goals and the instrument to discuss concerns and answer questions (such as a professional coach). Scores on scales, and particularly on questions, can fluctuate, even though a person’s “true” score remains the same. Comparing scores (question by question or scale by scale) to measure change over time or to measure the impact of an initiative or other leadership development experience may provide misleading information. CCL’s experience with 360-degree retests has demonstrated the difficulty in accurately measuring behavioral change using this method (see Exhibit 3.4). Climate Survey Retests When leadership development is a component in an organization’s efforts to make significant improvements or changes to its work climate, it can be useful to examine the extent to which that climate has changed over the course of the development initiative. This can be accomplished by making at least two assessments of the organization: one before the beginning of the initiative and one at an appropriate time after the initiative has ended. Many organizations conduct
Designing and Conducting the Evaluation
c03.indd 45
45
2/1/08 3:53:32 PM
Exhibit 3.4. How CCL Measures Change CCL typically uses a change survey called Reflections. This survey is designed to measure behavioral change using a “retrospective pretest and posttest” design. (This design is discussed on page 44.) Reflections is a 360-degree follow-up assessment that provides feedback to individual participants regarding their leadership growth and, in an aggregate form, feedback to organizations, to community groups, and to CCL regarding the impact of an initiative on a group of participants and their organizations. To learn more about Reflections, see Martineau (2004) and Exhibit 3.12. Contact CCL for more information.
annual climate surveys as a general practice. The appropriate time to administer subsequent assessments depends on the type of climate change expected, the size and complexity of the organization, and the number of individuals who have participated in the initiative. Change in larger, more complex organizations takes more time than in smaller, less complex situations. Organizational climate is typically defined by employees’ satisfaction with specific features, such as pay and benefits, leadership, and opportunities for development. As with other measures of organizational improvement, climate change should be examined only if the leadership development initiative was designed to effect this type of large-scale change. The initiative should include a sufficient number of employees (the appropriate number depends on the type of initiative and the change desired), should be of long enough duration to create change at the organizational level, and should be designed to encourage organization-level changes. There are many factors beyond leadership development that can have an impact on climate. Culture Survey Retests Culture is typically defined as the behaviors and values that pervade an organization or group. It determines, among other things, whether a particular behavior, appropriate in one organization, is deemed inappropriate in another. The culture
46
c03.indd 46
Evaluating the Impact of Leadership Development
2/1/08 3:53:33 PM
of a group or organization is built by its people and is extended through its selection, attrition, reward, and reprimand processes. Culture is greater than any one individual’s values or behaviors. When leadership development is a component in an organization’s efforts to make significant improvements or changes to its culture, it’s useful to examine the extent to which that culture has changed over the course of the leadership development initiative. You can make this part of your evaluation by making at least two assessments of the organization: one before the beginning of the initiative and one or more at appropriate intervals after the initiative has ended. The appropriate time to administer subsequent assessments depends on the type of culture change expected, the size and complexity of the organization, and the number of individuals who have participated in the initiative. Additional administrations of assessments, including administrations during the initiative, provide stronger evidence of impact. Culture is more embedded than is climate, and so is more difficult to change. Assessments of culture change are unlikely to reveal significant data more than once a year. As with organizational climate change, the success and speed of culture change depend largely on the size and complexity of the leadership development initiative, as well as on the position and percentage of employees involved in the initiative. As with other measures of improvement, culture change should be examined only if the leadership development initiative was designed to effect this type of large-scale change. The initiative should include an appropriate number or percentage of employees from organizational levels capable of effecting the culture change desired, should be of a duration long enough to create change at the organizational level, and should be designed to encourage organization-level changes. Measuring Return on Investment (ROI) Although return on investment is a phrase that most often refers to financial implications, it can also describe the broader impact of individual and team development initiatives on an organization. Several methods are available to evaluate such impact. Determining which is the most appropriate for a particular evaluation depends on the type of impact expected (see, for example, Chapter 5 of The Handbook of Leadership Development Evaluation [Hannum, Martineau, &
Designing and Conducting the Evaluation
c03.indd 47
47
2/1/08 3:53:33 PM
Reinelt, 2007]). To identify the appropriate measure at the organizational level, some useful questions to ask include the following: • What changes are expected as a result of the initiative? • Why are these changes expected? (In other words, what is the connection between the objectives of the leadership development experience and these changes?) • When can stakeholders in the organization or in the community expect to see changes? • Who will be able to note and report on these changes? • How can data about these changes be obtained? True measures of ROI include data such as the costs of facilities, trainers, materials, and the time participants spend in training and away from their jobs. ROI formulas also include the financial benefits of training, such as cost savings, new revenue, and calculations of the value of perceived job improvement. A critical step in creating accurate ROI formulas is isolating and measuring the effects of development. Although ROI formulas work well for skills-based training and many organizations apply them to leadership development, the value of this method is limited. It doesn’t provide data related to the quality of improvement. For example, an ROI of 300 percent is impressive, but without data showing where the improvement has been made, it’s not possible to fully measure the impact of that improvement.
DESIGNING THE EVALUATION PLAN Organizations and foundations benefit from an evaluation plan because it clarifies what is happening, when, and why. As you plan your evaluation, be sure to indicate the relationship between specific evaluation questions, components of the leadership development initiative, timelines, and selected evaluation methods. The evaluation plan design examples in Chapter 3 can serve as templates for a plan that serves your particular circumstances. At the time you create your evaluation plan, it’s not necessary for you to have identified specific content for each method (for example, the questions for your survey). Your intent at this point should be to choose methods that are likely to produce the type of data valued by key stakeholders, capable of addressing each specific evaluation question, assigned at an appropriate time, appropriate for the group providing the data, and logistically feasible. Your plan provides an overview of datacollection activities and helps ensure that you are collecting the appropriate data
48
c03.indd 48
Evaluating the Impact of Leadership Development
2/1/08 3:53:34 PM
needed to answer the evaluation questions. A good evaluation plan will include other activities, such as the communication of results (for example, what results are communicated, to whom, and by what media). Elements of an Evaluation Plan An evaluation plan is a way to document evaluation rationale and processes from a very high level in order to make sure the various elements of an evaluation make sense and fit together as intended. The plan is also a way to make sure stakeholders understand how and why an evaluation is being conducted. Evaluation plans can vary, but you will probably want to include the following elements in your plan: • A title that clearly indicates what is being evaluated. For example, the title might be “Longitudinal Evaluation of the Implementation and Outcomes of the ABC Leadership Development Program.” • A list of evaluation questions. The evaluation plan should capture the major questions being investigated, not the specific questions to be used in data collection. • An indication of the stakeholder groups from which data will be gathered and with whom information needs to be shared. For example, “Program participants will contribute to the data for questions 1, 2, 3, and 5; program sponsors for questions 2, 4, and 5; and participants’ managers for questions 1, 3, and 4. Respondents will see a summary of the questions to which they contribute data, and program sponsors will see a summary and details of all data collected.” It is critical that the evaluation make clear which major questions will be asked of which stakeholder groups and who will be seeing the various pieces of data so that questions of intent and completeness do not arise later. • An indication of how data will be gathered. Will data be gathered by surveys, interviews, or other methods? Specifically noting the means of data collection will keep stakeholders apprised of what will be collected, how, and when. • A timeline for data collection and any important communication activities. Identify what data will be collected before, during, immediately after, and several months and/or years after the program has taken place, and when interim and final reports will be shared with key stakeholders. Evaluation plans can be as simple or as complex as you need them to be. See Tables 3.1 and 3.2 for two examples illustrating the different ways information can be displayed. Designing and Conducting the Evaluation
c03.indd 49
49
2/1/08 3:53:34 PM
50
c03.indd 50
Evaluating the Impact of Leadership Development
2/1/08 3:53:35 PM
What changes are resulting in participants’ work groups?
Survey (P); Interviews (P)
Are participants receiving support for development and application of new competencies?
Survey (P)
How do participants intend to apply their learning and new skills? 360-degree behavioral change instrument (P, M, DR, Pe, O)
Survey (P)
Were learning objectives met?
April
Is there growth in competencies?
January
Evaluation Questions
Survey (P, M); Interviews (P, M)
August
November
Survey (P); Interviews (P)
Time Frame and Data-Collection Methods (Source*)
Table 3.1. Evaluation Plan Example 1 Longitudinal Outcome Evaluation for the ABC Leadership Development Program
Designing and Conducting the Evaluation
c03.indd 51
51
2/1/08 3:53:35 PM
February: mini summary June: 360-degree (S, C) reports (P), July: interim report (S, C)
C key organizational client contact
DR direct report of participant
O other (client, customer, etc.)
M manager of participant
Pe peer of participant
P participant (individual, group, or team)
*Data Source Key
Communication Timeline
How is the organization benefiting? September: mini January: final, summary (S, C) comprehensive report (S, C)
Analysis of organizational data (C)
52
c03.indd 52
Evaluating the Impact of Leadership Development
2/1/08 3:53:36 PM
O N L Y
P H A S E
P I L O T
Data-Collection Timing
immediately after focus groups the program immediately after the program
Are all aspects of the program functioning as intended (in order to meet objectives)?
focus groups
Participants
Is the flow of the program logical and helpful to participants?
Why do participants select this program? What appeals to them about the program? What doesn’t appeal to them?
Do program staff and faculty have the during and information and resources they need immediately after to provide the highest-quality prothe program gram? What’s helpful? What’s missing?
during and How effective is the current logistical immediately after support? How well are program the program processes (advance information for participants, test administration process, program set-up, etc.) functioning in the multiple locations?
Questions
program debrief
program debrief
program debrief
program debrief program debrief
program debrief
Coordinators
Trainers
Table 3.2. Evaluation Plan Example 2 Pilot and Ongoing Implementation and Outcome Evaluation of the MNQ Leadership Development Program
focus groups
Sales Staff
Designing and Conducting the Evaluation
c03.indd 53
53
2/1/08 3:53:36 PM
E V A L U A T I O N
S T A N D A R D three months after the program
360-degree retrospective pre/post survey
immediately after end-of-program the program survey
immediately after end-of-program the program survey
What is the impact on the participants’ three months and 360-degree retrospective organizations? six months after pre/post survey the program and impact survey
What are the changes in the selected competencies?
To what extent are the program objectives relevant?
To what extent does the program meet the stated objectives?
• tailored summaries of lessons learned to participants and coordinators one month after the program
• tailored summary of lessons learned to sales staff one or two days after presentation to program design team
• written report and presentation to program design team (including trainers) two weeks after the program;
Pilot Phase Communication Plan:
(Continued )
54
c03.indd 54
Evaluating the Impact of Leadership Development
2/1/08 3:53:37 PM
P R O C E S S
• semiannual impact survey report to program manager
• semiannual aggregate 360-degree report to program manager;
• individual 360-degree reports to participants the day after datacollection deadline;
• quarterly summary reports of end-ofprogram survey data to program manager;
• end-of-program survey to trainers and coordinators immediately;
Ongoing Communication Plan
impact survey
What are the barriers and supports for six months after making changes? the program
Participants impact survey
Data-Collection Timing six months after the program
Are there any unintended consequences of the program?
Questions
Table 3.2. (Continued ) Trainers
Coordinators
Sales Staff
The sample plan in Table 3.1 indicates that surveys, a 360-degree assessment, interviews, and organizational data will be used in January, April, August, and November to collect data relevant to the various questions listed in the far-left column. For some evaluation questions, data from multiple perspectives are gathered and multiple data-collection strategies are used. The data source key allows you to indicate where or from whom the data are being collected.
DATA - COLLECTION METHODS There are many tools and techniques evaluators can use to gather data. The challenge is to select the appropriate tools and techniques to efficiently and effectively fulfill the purposes of the evaluation. In the following pages we describe several techniques that are particularly relevant to evaluating leadership development initiatives. Most, if not all, of these methods are useful for evaluating other types of programs and initiatives as well. We also recognize that there are other methods that other evaluators and practitioners may use for evaluating leadership development programs, depending on the form those programs take, the outcomes targeted, whether the program is internal or offered by an external provider, and the extent to which the program is tied to other improvement efforts that offer their own data. What is important is that you identify the methods that will produce the type of data your stakeholders expect, in the time frame they expect it, and by collecting data in a way that is accepted by stakeholders. As you consider different methods, you will also need to think about administration tools (for example, how survey data will be gathered) and storing your data (for example, creating a database). Do you have the resources to fully implement and use the data from the methods you have selected? For instance, if you are gathering interview data, do you have the resources to transcribe the interviews? For both quantitative and qualitative forms data, do you have the resources to analyze the data in an appropriate manner? Descriptions, examples, and advice follow. General information about reporting evaluation findings is included in Chapter 4. Expectations Survey Gathering information about participant expectations before an initiative can be a helpful way to make sure participants clearly understand the initiative in which they are participating and to document their preprogram perspective with a goal
Designing and Conducting the Evaluation
c03.indd 55
55
2/1/08 3:53:37 PM
to follow up after the program to compare. Information from an expectations survey provides initiative designers and trainers with information about whether or not participant expectations differ from initiative objectives. They can also compare objectives with what participants perceive as benefits (if the survey asks about perceived benefits or relevance). An example is shown in Exhibit 3.5. Advice
• Develop quantitative survey questions that assess participants’ perceived needs regarding each of the expected objectives. Then allow some space for participants to write in other expectations they may have. • Ask about the perceived benefit or relevance of the initiative. These data can be compared to post-initiative data in order to determine shifts in perspective. These data can also indicate to trainers the extent to which the rationale for having the initiative needs to be shared with participants. • It is generally a good idea to share survey information with program staff before the initiative. • Participants are sometimes interested in seeing these data early in their initiative. Sharing these data with participants can be a way for program staff to
Exhibit 3.5. Preprogram Expectations Survey Example To what extent does each of these developmental areas represent a goal on which you want to work during this leadership development initiative? Use the scale below for your ratings. Not at all 1
To some extent 2
3
To a great extent 4
5
—— Improve Self-Awareness Learn how others perceive you Understand how your management style impacts those with whom you work Be aware of the impact of your behavior on others (Continued)
56
c03.indd 56
Evaluating the Impact of Leadership Development
2/1/08 3:53:38 PM
—— Improve Self-Confidence Be self-assured Believe you can make valuable contributions to the organization Recognize and appreciate your talents and abilities ____ Build and Maintain Relationships Be approachable and receptive to others Support and understand the needs of others Display patience with others in difficult situations Avoid being abrasive with others ____ Work Across Organizational Boundaries Consider the impact of your actions on the entire system Balance what is good for the unit or area with the needs of other parts of the organization Deal effectively with contradictory requirements or inconsistencies in the organization Evaluate the organization in a clear and objective manner [And so on, until all objectives have been addressed.]
explain the connection between the initiative and participants’ expectations and to address any misconceptions or “disconnects.” Daily Evaluation In some leadership development initiatives, participants complete evaluation forms at the end of each day. The value in these forms is two-fold. First, they give participants an opportunity to reflect on their daily experiences, which reinforces what they’ve learned (especially when questions are written in a way that requires reflection). Second, they provide program staff with information that enables them to make immediate adjustments to the initiative, thereby enhancing its effectiveness. This method is somewhat limited in that it does not offer participants much time to reflect on their experiences and may not provide a comprehensive picture of the experience. Exhibit 3.6 provides an example. Designing and Conducting the Evaluation
c03.indd 57
57
2/1/08 3:53:38 PM
Exhibit 3.6. Day 1 Evaluation The daily evaluation process is designed to provide a source of reflection for you and provide the staff with feedback about the program. Your candid responses are important and appreciated, especially your additional written comments. Not at
Very
All
Great
Today’s Experience Please rate today’s session in terms of
1
2
3
4
5
1
2
3
4
5
the value of what you learned. Please rate today’s session in terms of the relevance of what you learned. What was the most helpful thing you learned today? Why?
your learning?
Not at All
Communicating Your Vision
1
2
3
4
5
Mind Mapping
1
2
3
4
5
Stop and Reflect
1
2
3
4
5
Other: _______________________________
1
2
3
4
5
Which components contributed most to
Very Great
What, if anything, should have been added to today’s session?
What do you want the faculty to do differently tomorrow?
What recommendations do you have to improve the session?
58
c03.indd 58
Evaluating the Impact of Leadership Development
2/1/08 3:53:38 PM
Advice
• Keep your form as short as possible; participants are usually tired at the end of the day. • Ask only for information you need at that point in time. It can be helpful to first determine what information program staff will actually be able to use— what immediate changes are truly realistic. • Use paper or an electronic format to gather data. If you use paper, make sure to have a few extra copies on hand and writing utensils for participants to use. An electronic approach will require enough computer stations or hand-held devices for each participant. • Consider whether or not you want participants to put their names on the forms. They may be more comfortable being frank if they do not have to give their names. However, you would not be able to link the information from the survey to any other information from that person, nor will you be able to follow up with someone for further clarification or to address a specific issue. • For the data to be helpful in the moment, you may want to schedule time at the end of the day to review the forms and communicate information to program staff (verbally or through a very short summary highlighting any concerns as well as anything that went particularly well).
End-of-Initiative Evaluation Participants can complete end-of-initiative evaluation forms at the conclusion of each component of the leadership development initiative. You can design these forms to capture how well a specific component met its target objectives, how participants intend to apply what they’ve learned in the workplace, and how well facilitators, facilities, and logistics met a specified standard. Use these forms to gather evidence regarding how participants intend to use lessons learned, to collect impressions of how relevant and valuable the initiative is to potential participants, and to capture suggestions for changing the initiative. Capturing this information while it is fresh on the minds of participants is helpful, but this method doesn’t measure the actual implementation of the intended changes—only the intent to apply what has been learned. Exhibit 3.7 provides an example.
Designing and Conducting the Evaluation
c03.indd 59
59
2/1/08 3:53:39 PM
60
c03.indd 60
Evaluating the Impact of Leadership Development
2/1/08 3:53:39 PM
2
3
4
5
1 Not at all
program. 2 To a little extent
3 To some extent 4 To a great extent
5 To a very great extent
Please circle the numbered rating that reflects the extent to which each outcome was achieved for you during the
Program Outcomes
Comments: Please comment on any aspect of your program experience that you would like to see changed.
Comments: Please comment on any aspect of your program experience that you found particularly helpful.
1
Please rate your overall satisfaction with the program.
Very great
4 To a great extent 5 To a very great extent
Not at all
2 To a little extent 3 To some extent
Overall Program Evaluation
Program Overall
1 Not at all
Please use the following rating scale to evaluate this program:
Your Name: _______________________________________ Program Date: ______________________________________
feedback on your experience.
Maintaining program quality is important to us. Please complete this evaluation form to provide us with helpful
Exhibit 3.7. Final Program Evaluation
Designing and Conducting the Evaluation
c03.indd 61
61
2/1/08 3:53:40 PM
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
The Outcome Was Achieved
Personal Impact: How have you benefited from your attendance in this program?
Maintain composure under stress
Managing Yourself
Coach others to perform at their best
Developing Others
Build and maintain effective working relationships
Building and Maintaining Relationships
b. Adapt to change easily
group
a. Get people aligned behind change in my
Change and Adaptability
Remain open to feedback from others
Giving and Receiving Feedback
Demonstrate a respect for varying backgrounds and perspectives; value cultural differences
Differences Matter
Be aware of the impact of my behavior on others
Increasing Self-awareness
As a result of this program, I am better able to:
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
5
5
5
5
5
5
5
5
(Continued )
4
4
4
4
4
4
4
4
The Outcome Is Applicable
62
c03.indd 62
Evaluating the Impact of Leadership Development
2/1/08 3:53:41 PM
Yes
No
Yes
No
provoking manner.
Clarity of Presentation: Transmits contents (models, topics, data, etc.) in an understandable, engaging, and thought-
criteria:
Please rate each program facilitator’s ability to create an effective learning environment based on the following
Facilitator Impact
____________________________________________________________________________________________________________________________________________________________________________________________________________________
____________________________________________________________________________________________________________________________________________________________________________________________________________________
Is there anything else you would like to tell us?
___________________________________________________________________________________________________________
___________________________________________________________________________________________________________
Please comment: __________________________________________________________________________________________
Circle One:
Did you have the opportunity to work on other outcomes that were important to you?
__________________________________________________________________________________________________________
__________________________________________________________________________________________________________
Please comment: _________________________________________________________________________________________
Circle One:
or were not met?
Do you have other comments regarding the ways in which the outcomes listed on the other side of this form were
result of your attendance in this program?
Organizational Impact: What changes will you make personally in your group, business unit, or organization as a
Exhibit 3.7. (Continued )
Designing and Conducting the Evaluation
c03.indd 63
63
2/1/08 3:53:41 PM
2 2 2 2
1 1 1 1
Not at all
3
3
3
3
4
4
4
4
5
5
5
5
Very great
Clarity of Presentation
1
1
1
1
2
2
2
2
Not at all
3
3
3
3
4
4
4
4
5
5
5
5
Very great
Process Facilitation
1
1
1
1
2
2
2
2
Not at all
3
3
3
3
4
4
4
4
5
5
5
5
Very great
Credibility
Thank You!
Comments: What do your facilitators need to improve? (Please comment on program coordinators, if appropriate.)
coordinators, if appropriate.)
Comments: What did you find particularly effective/helpful about your facilitators? (Please comment on program
Facilitator Name
Credibility: Establishes competence, respect, and believability.
Process Facilitation: Recognizes and elicits learning and insight from experiences occurring in the classroom.
Advice
• Remember that participants have not had an opportunity to apply what they have learned, so the most you can hope to know about is their intent to use what they have learned. • If the end of the initiative is a face-to-face event, you may want to ask participants to complete the survey before they leave the event. Response rates are usually higher if you give them the time to complete the form on-site. On the other hand, participants may be tired and may provide more thoughtful and complete responses if they complete the survey later. • If there are numerous open-ended questions, consider administering the survey electronically to avoid having to retype responses. You may also want to consider a focus group or interview to gather these data. • Consider whether or not you want participants to put their names on the form. They may be more comfortable being candid if they do not have to give their names. However, you would not be able to link the information from the survey to any other information from that person, nor will you be able to follow up with someone for further clarification or to address a specific issue. • If the initiative has multiple cohorts, you may want to communicate information about what did and did not go well to the program staff for the next cohort so they can make changes if needed. Interview Interview questions are typically open-ended, they provide qualitative data (though you may ask a few quantifying questions as well), and they can be asked face-to-face, by videoconference, or by telephone. You can use interviews to assess perceptions of the initiative from a stakeholder’s perspective. You can also obtain examples of how behavior has changed and/or how what was learned has been applied and what changes have resulted. If resources don’t allow for one-on-one interviews, you may decide a focus group is a better option or adapt the interview format to an open-ended questionnaire that you can mail or e-mail to participants. You may want to follow up for more detail from selected respondents.
64
c03.indd 64
Evaluating the Impact of Leadership Development
2/1/08 3:53:42 PM
Interviews are most effective when you want qualitative information to fulfill multiple purposes. Such purposes include the following: • Assisting in identifying training and learning needs, an initiative’s design, or expectations for applying lessons from the development initiative (interviews would take place before the initiative); • Determining participants’ reactions, experiences, and satisfaction with the initiative (interviews would take place after the initiative); • Determining participants’ reactions, learning, and intentions to apply their learning (interviews would take place during an initiative or after it had ended); • Gathering information to help develop surveys, focus group interview questions, or the focus of an observation (interviews could take place at any point relative to an initiative, depending on the outcomes desired); or • Further interpreting survey results (interviews associated with evaluation are typically most relevant for this purpose after an initiative has been completed). Interviews have several advantages and disadvantages. One advantage is that interviews allow the evaluator to probe for clarification and deeper complexity, which results in richer data. They provide time for participants and stakeholders to reflect, which can be developmental in itself. A significant disadvantage is that interviews can be more costly than other methods. If the interviews are conducted in person, for example, travel costs add to the total price of the evaluation. If interviews take place over the phone, phone charges need to be considered. Interviews can also take more of the evaluator’s time, both in terms of conducting the interviews and analyzing the qualitative data. Exhibits 3.8 and 3.9 provide examples. Advice
• Remember that interviews can be conducted over the phone or by using Internet-based collaboration tools (such as electronic whiteboards or chat rooms), which will reduce travel costs and also allow evaluators to conduct more interviews in a shorter amount of time. However, it is much harder to build rapport and almost impossible to read nonverbal communication. If sensitive information is sought in the interview, it may be worth the extra expense of conducting face-to-face interviews to ensure that higher-quality data are provided.
Designing and Conducting the Evaluation
c03.indd 65
65
2/1/08 3:53:42 PM
Exhibit 3.8. Participant Interview
Participant Interview Interviewee: Interviewer: Date: 1. To what extent has your effectiveness as a leader within your organization been improved due to your participation in the program? 1
2
3
4
5
6
Not at All
7 Substantially
2. What aspects of your job have been influenced by your experience gained through the program? How were they influenced? 3. How clear were the expectations/objectives of the program? (for each get a rating and explanation of rating) For the face-to-face program in 1
2
3
4
5
6
Not at All
7 Completely
For participation in 1
2
3
4
5
6
Not at All
7 Completely
For your participation with your coach 1
2
3
4
5
Not at All
6
7 Completely
4. What specific aspects of the coaching process led to your meeting the objectives you set for yourself in the program? (Continued)
66
c03.indd 66
Evaluating the Impact of Leadership Development
2/1/08 3:53:43 PM
5. What did your coach do that was particularly helpful? (Note to interviewer: Push for specific behaviors as well as more intangible characteristics such as credibility, empathy, etc.) 6. What did your coach do that was not helpful or fail to do that would have been helpful? What would you like to see done differently? 7. Overall, how satisfied are you with your progress at this point in time? 1
2
3
4
5
6
Not at All
7 Completely
8. What has contributed to or detracted from your satisfaction? 9. How was the percentage of time you spent with your coach balanced between discussing individual and organizational issues? (for example, 20:80, 50:50, . . . ) 10. What kinds of organizational issues were discussed? 11. What kinds of individual issues were discussed? 12. How useful was the coaching component of the program? 1
2
3
4
Not at All
5
6
7 Extremely
13. What was or was not useful? 14. Did you receive written feedback from your coach? If so, how was it helpful (or not)? 15. What changes would you recommend to the program overall?
• When scheduling interviews, let interviewees know approximately how long the interview will take and, if appropriate, provide them with the questions in advance. Also let them know how data will be treated and reported (whether, for example, their comments will be confidential). • If multiple people will be conducting interviews, review the protocol together to be sure everyone understands what kind of follow-up questions might be needed so that consistent data are collected.
Designing and Conducting the Evaluation
c03.indd 67
67
2/1/08 3:53:43 PM
Exhibit 3.9. Executive Sponsor Interview Evaluation of Questions for Interviews with Senior Executive Sponsors of the Initiative 1. What kinds of leadership challenges were your participants aiming to
address through this program? In general, and for each of your participants (without having to use names) if they vary by person. (What did you see as the overall goal of the initiative? What were you hoping the participants would get out of it? What are their leadership challenges? What were you hoping your organization would get out of it?) 2. What have you observed regarding the impact of the
Name> on the participants who report to you and their teams? Did you see improvements in the leadership challenges they were facing (mentioned above)? 3. What have been the most significant changes you have observed in
the participants of the ? Probe if you don’t hear specific examples: Can you give some concrete examples of the kinds of changes you have observed? (Make sure interviewee is o.k. with this example being shared in a group report.) Again, bring the discussion back to changes or improvements related to leadership/ management challenges that this person hoped the program would help address with the participants. 4. What, if any, impact have you observed on the organization as a
result of this program? 5. What, if any, barriers do you see the participants facing in their
ability to make changes in their leadership effectiveness after participating in the program? 6. In what areas do you think the participants require additional
development to become more effective leaders? 7. Is there anything else that you feel is important to share about the
impact of the ?
68
c03.indd 68
Evaluating the Impact of Leadership Development
2/1/08 3:53:43 PM
• Consider having the evaluator conduct the interviews (as opposed to the organization’s HR department, for example). Acceptance and endorsement for the evaluation can increase, because participants and stakeholders often view the evaluator as an objective party. • You may want to specify the type of location in which the interview should be conducted. This is especially important if interviewees do not have private workspaces. • If the data are confidential, you must remove any names or titles used in the interviews. You may also want to remove specific examples used in the interviews if the examples make it clear who provided them. Learning Survey A learning survey is designed to assess the extent to which participants have learned new content during the initiative. This method is valuable when participants are expected to retain factual information (such as their organization’s leadership or competency models or its business policies or practices) or learn specific steps for implementing leadership responsibilities (such as models for giving feedback and coaching others). There are generally two ways to assess the attainment of factual information. One is to administer a learning survey after the initiative. The other is to administer the survey twice: once before the initiative and again immediately afterward. The second approach has two advantages over the first. The pretest, a survey before the initiative, provides a means of assessing the needs of participants: this can guide you in focusing the measures you will use in the evaluation. This approach also allows you to create a logical tie to what has been learned with the initiative. The learning survey’s questions must be related to the content of the initiative in order to be a measure of what participants have learned. See the example in Exhibit 3.10. Advice
• Focus on the content that is to be learned, rather than on broader lessons that participants may have learned. Just as teachers and professors will design classroom tests to measure students’ knowledge of a given area, the learning survey should be designed to assess knowledge gain.
Designing and Conducting the Evaluation
c03.indd 69
69
2/1/08 3:53:44 PM
Exhibit 3.10. Learning Survey (Example) Potential Learning Survey Questions What are the eight competencies necessary for leadership success in ABC Organization? (List all eight.) What are the primary ways in which leaders fail at ABC Organization?
What are the primary components of the “Success for Leadership” model?
How are ABC Organization’s competencies related to the “Success for Leadership” model? (Draw a diagram.)
What are the steps in an effective decision-making process?
What are the steps in the “Coaching Others” process?
What activities are leaders expected to implement as part of ABC Organization’s succession-planning process?
What is the chain of command for decisions related to new-project initiation at ABC Organization?
Which groups are leadership teams at ABC Organization expected to include in decisions related to performance improvement and measurement practices?
70
c03.indd 70
Evaluating the Impact of Leadership Development
2/1/08 3:53:44 PM
• If you are interested in participants’ thoughts about application of what has been learned, either frame the questions as additional information or use another method (such as the end-of-program survey) to collect these data. While it is perfectly acceptable to use a learning survey to measure depth of knowledge, you want to be careful that participants do not feel tested in areas in which they should not be expected to have increased their knowledge. • Learning surveys are generally intended to measure how much participants have learned about a specific topic, model, method, etc. If the survey is openended, provide a summary of how many participants have learned the expected information. If it has a multiple-choice format, provide frequencies of correct responses. • Provide recommendations for which components need more attention— where learning did not reach the intended levels. Change Survey Change surveys are useful in assessing whether change has occurred as a result of an initiative. They are typically used to measure changes in attitudes or behaviors specific to the initiative. A well-developed change survey should be based on what is already known about the impact of the initiative and/or the objectives of the initiative. Evaluators should ensure that the response scale used in the change survey facilitates the measurement of change (see “The Measurement of Change: Methods and Issues” on pages 43 to 48). Change surveys are most effective when the focus of the evaluation is on behavioral changes as measured by quantitative data. As discussed previously, there are various ways to measure change. On the following pages, we provide examples of three different ways. The first (Exhibit 3.11) is a basic change survey, which is useful if you only need to know about the degree and direction of changes following a leadership development initiative. This approach does not indicate participants’ abilities or knowledge before the initiative or whether more change is needed. For instance, you could determine that there has been a great deal of positive change in selfawareness, but not whether more training or support would be required to reach the desired outcomes. The second example (Exhibit 3.12) is a 360-degree retrospective before-now survey. This approach provides a wealth of information, including the participants’ abilities before the initiative and how they are performing after the initiative. It allows you to determine the amount of change and—because it measures
Designing and Conducting the Evaluation
c03.indd 71
71
2/1/08 3:53:44 PM
Exhibit 3.11. Change Survey Example 1 Using the rating scale provided below, please indicate (by writing in the appropriate number to the left of the item) the extent to which you have changed in the indicated areas. 1 considerable change for the worse 2 moderate change for the worse 3 a little change for the worse 4 no change 5 a little change for the better 6 moderate change for the better 7 considerable change for the better _____ Improved Self-Awareness Learn how others perceive you Understand how your management style impacts those with whom you work Be aware of the impact of your behavior on others _____ Improved Self-Confidence Be self-assured Believe you can make valuable contributions to the organization Recognize and appreciate your talents and abilities _____ Building and Maintaining Relationships Be approachable and receptive to others Support and understand the needs of others Display patience with others in difficult situations Avoid being abrasive with others _____ Working Across Organizational Boundaries Consider the impact of your actions on the entire system Balance what is good for the unit or area with the needs of other parts of the organization Deal effectively with contradictory requirements or inconsistencies in the organization Evaluate the organization in a clear and objective manner [And so on, until all objectives have been addressed.]
72
c03.indd 72
Evaluating the Impact of Leadership Development
2/1/08 3:53:45 PM
Exhibit 3.12. Change Survey Example 2
current performance—you can determine whether more change is needed in specific areas. However, it can be time-consuming to administer, and it relies on the raters’ ability to accurately remember behaviors—meaning that you can’t wait too long to administer it. The third example (Exhibit 3.13) is a survey that requests participants’ current state and the amount of improvement they have made as a result of the initiative. This approach provides information similar to that of the retrospective before-now survey and has similar drawbacks. Guidelines for writing survey questions are provided in Exhibit 3.14. Valid, reliable change surveys can be time-consuming to develop, but they can be relatively inexpensive to administer. They allow for responses from a large sample of people and can collect easily analyzed quantitative data. They allow for anonymity or confidentiality of responses, are useful when respondents are geographically dispersed, allow respondents to respond on their own time, and Designing and Conducting the Evaluation
c03.indd 73
73
2/1/08 3:53:45 PM
Exhibit 3.13. Change Survey Example 3 Please rate the participant on his/her goal(s) on two dimensions: how effective he/she is currently, and how much improvement you have noticed over the past six months.
Goal 1
Current Effectiveness not at all
1
moderately
2
3
4
5
6
Amount of Improvement very
none
7
1
moderate
2
3
4
great
5
6
7
Here are specific behaviors the participant has been working on. Please rate to what extent he/she is effective now and how much improvement you have noticed over the past six months. Current Effectiveness not at all 1
moderately 2
3
4
5
6
Improvement
very
none
7
1
moderately 2
3
4
5
great 6
7
1. 2. 3. 4. [And so on, until all goals have been addressed.]
require that all respondents answer the same set of questions. On the minus side, they don’t allow for changes to or clarification of questions. Intuition can be misleading when developing a survey. Even seemingly simple choices, such as what response options to use, can have an unintended impact on results. If you plan to develop your own survey and do not have training in survey development, it is wise to seek advice from measurement or psychometric publications or from measurement professionals. As with the other data-collection methods, you would probably benefit from using the pilot study process described in Exhibit 3.1.
74
c03.indd 74
Evaluating the Impact of Leadership Development
2/1/08 3:53:46 PM
One way to determine the changes that have occurred is by measuring all participants according to the intended outcomes of a given program. An example of this format appears below. This format requires one rating of change for each intended outcome. This format provides only an “amount of improvement” rating. A second way to measure change at the program level is to use a “Retrospective Before–Now” rating format. An example of this format appears in Exhibit 3.12. For this type of change measure, two ratings are collected for each intended outcome. This type of format provides a “current effectiveness” rating and allows for an “amount of improvement” rating (by subtracting the “Before” scores from the “Now” scores). Another way to evaluate change is to customize the survey to the goals of each individual participant. An example of this approach appears in Exhibit 3.13. This type of change measure provides both a “current effectiveness” and “amount of improvement” rating. Advice
• Determine what specific type of change you are looking for or expecting to find, and then use the most appropriate question types to collect the data (frequency, effectiveness, etc.). • Determine the most appropriate timing to measure the change you are looking for. Some types of change can be assessed soon after a program has been completed, while others may take up to a year to become measurable. • Make sure response scales measure the degree of change expected and allow participants the appropriate amount of variability of choice. • Depending on the type of questions and response format used, data may be reported as frequencies, means, difference scores, or in some other way. Providing the most information and meaning possible from whatever data are collected is critical to being able to communicate the changes that have been made.
Behavioral Observation Behavioral observation involves observing a set of activities, the people who are participating in those activities, and the environment in which the activities take place. Observations can produce qualitative data in the form of field notes or quantitative data if observers note their information as ratings, rankings, or frequencies.
Designing and Conducting the Evaluation
c03.indd 75
75
2/1/08 3:53:46 PM
Exhibit 3.14. Guidelines for Writing Survey Questions Surveys are frequently used to collect information from a large number of individuals on a broad range of subjects. Although commonly used, a survey is one of the most challenging instruments to develop well. When designed in haste, surveys often collect inaccurate and useless information. Here are a few simple guidelines for developing survey questions. Before you begin the question-writing process, clearly identify the areas you want to assess. Select representative behaviors, skills, and competencies. Relate them to the initiative and prioritize them. Then determine the type of information you need to capture. For example, you may need to collect data related to the frequency of a specific behavior or related to the evidence that a defined skill level has been attained. This process will help you write survey questions that are specific and relevant to your evaluation.
Things to Do When Writing Survey Questions • Be as concise as possible without losing the meaning of the question. • Use specific language with common meaning and interpretation. • Create several questions to measure complex phenomena. • Write open-ended questions that ask for specific information. • Get feedback on the questions from people with different and relevant perspectives.
Things Not to Do When Writing Survey Questions • Don’t use colloquialisms, metaphors, similes, figures of speech, culturally biased phrases, scientific words, or jargon. • Don’t be unnecessarily wordy. • Don’t use double negatives. • Don’t use language that is potentially inflammatory or derogatory. • Don’t attempt to measure several ideas in a single question.
76
c03.indd 76
Evaluating the Impact of Leadership Development
2/1/08 3:53:47 PM
There are several effective uses for behavioral observation. Before the initiative begins, observation can determine participants’ baseline knowledge, skills or behaviors, and attitudes. During the initiative it can determine levels of interaction, engagement, skill development, and satisfaction with the initiative. Observations conducted between one week and three to six months after completion of the initiative can determine changes in knowledge, skills or behaviors, and attitudes. Behavioral observation is especially effective when it is used as one of several data-collection methods. Compared with other evaluation methods, behavioral observation has several advantages. For example, data are collected where the activity is taking place, thus enhancing the data’s validity; target activities are viewed within a context that may help interpret data collected from other methods; a trained observer might see things that others close to the initiative may miss; and the observation process can illuminate issues that interviewees are unwilling to talk about. This evaluation method also has some limitations. It requires well-trained observers, and training takes time. Multiple observers may focus on different things, thus making analysis and synthesis more difficult. Also, participants may alter their behavior if they know they are being observed. Finally, behavioral observation can be disruptive to the work environment and can be expensive if a large sample is required, and it can be time-consuming to gather and analyze the data. Exhibit 3.15 provides an example of a completed behavioral observation. In this example, directions for the observer are followed by possible responses, which appear in italics. In these directions, participant refers to the individual who has attended the development initiative being evaluated. Behavioral observation can also be quantified to a certain extent, providing data within a table that allows frequencies to be calculated. This way of recording behavioral observations is shown in Exhibit 3.16. Advice
• Be sure that behavioral observation will provide the data you are seeking. Given the time investment, it is an expensive method to conduct, and you want to be sure you are receiving your value out of it. Because the targeted development of leadership is often observable through natural (versus simulated) interactions with others, observers may sometimes interfere with what would otherwise occur.
Designing and Conducting the Evaluation
c03.indd 77
77
2/1/08 3:53:47 PM
Exhibit 3.15. Behavioral Observation: Qualitative Data (Example) Describe behaviors that indicate that the participant is accepting feedback from others. Pat listened as Fred explained that her behaviors made him feel uncomfortable in the meeting. Pat thanked Fred for the feedback and told him how she would try to change those particular behaviors in the future. Pat asked Fred to continue to feel comfortable in sharing his feedback with her. Describe behaviors that indicate that the participant is effectively delivering feedback to others. Pat told Mary that her presentation was well received by the client but needed to be tightened for the next presentation. Pat gave Mary suggestions for tightening her presentation. Pat made herself available to Mary for future questions. Describe behaviors that indicate that the participant is working effectively across organizational boundaries. Pat negotiated with the materials development director regarding a new product. Pat and the materials development director worked together to resolve the issue. Pat and the materials development director together presented the solution to senior management.
• Be as unobtrusive as possible as an observer. • Ensure that you have permission to observe. You should not make observations without the permission of those being observed. • Reach agreement with the people being observed that they are allowed to ask you to stop observing should a situation arise that is uncomfortable or
78
c03.indd 78
Evaluating the Impact of Leadership Development
2/1/08 3:53:47 PM
Exhibit 3.16. Behavioral Observation Rating Table: Quantitative Data (Example)
Behavior
Time of Occurrence
With Whom?
Accepted feedback from others
10:35 A.M.
Fred, Joan
to 1:15 P.M. Provided feedback to others
2:15 P.M.
Mary
Showed confidence when speaking
10:35 A.M.
Fred
Recognized others’ performance
2:15 P.M.
Mary
Gave constructive criticism to others
1:15 P.M.
Joan
Delegated authority to others
9:25 A.M.
Pete
confidential—especially if it is unrelated to changes expected from the leadership development efforts. • When possible, present the results in a confidential way. Do not disclose the individual or group being observed when you provide examples of your observations, unless it is appropriate and those involved are aware of how their data are being used. • Present the observations in terms of themes. Include examples and, if appropriate, frequencies to represent the number of times you observed a particular behavior of interest. Focus Group Focus groups usually include six to ten people. The primary purpose of this method is to obtain qualitative information from a team or group of individuals who have had a similar experience (participation in a training program, for example). Evaluations usually make use of multiple focus groups and use a welldesigned guide to focus the discussion. Focus groups should be carried out in a way that allows participants to feel safe disclosing information about their attitudes and perceptions regarding the initiative being evaluated. Focus groups can Designing and Conducting the Evaluation
c03.indd 79
79
2/1/08 3:53:48 PM
take place face-to-face, over the phone, or online. There are excellent resources for evaluators considering using focus groups (for example, see Greenbaum, 1999; Morgan, 1993; Morgan & Krueger, 1997). There are several ways you can use focus groups in your evaluation. Before an initiative begins, you can use focus groups to identify training or learning needs, to determine an initiative’s design, or to assess expectations of how participants will apply what they learn. During an initiative (assuming the initiative takes place over several days, weeks, or months), you can use focus groups to determine participants’ reactions to and experiences and satisfaction with the training initiative. After the initiative ends, you can determine participants’ reactions, what they’ve learned, how they intend to apply their new skills and knowledge, and the relative success or challenge they’ve had in doing so. Focus groups are also effective when it’s necessary to assess reactions, learning, and intentions of a team rather than individual participants. Some evaluators use focus groups to develop survey questions or to further interpret survey results. Another interesting product of focus groups is that they allow participants to process their experiences together, which can help them build support networks to further aid in their development. Focus group data collection has several advantages. Because it captures the collective experience of individuals, interaction among participants tends to increase the number and quality of responses. It provides a forum for teams to create additional impact and meaning related to their developmental experience. It lets the interviewer probe for clarification, explanations, and examples. Participants generally enjoy being part of a focus group interview, resulting in high response rates. This method is relatively inexpensive and makes good use of time, in that it allows evaluators to collect data quickly from a large group of people. The method also has limitations. A skilled facilitator is required to ensure that the data collected are of high quality. The facilitator has less control in a group than with an individual and so must have the skills and ability to keep the group on track. Data collected from a focus group may be difficult to capture and organize. Groups vary widely—some groups may develop a collective energy and provide extensive data, but other groups may lack energy and provide only superficial data. Exhibit 3.17 is an example of an outline for a focus group for participants who attended a program at a leadership development center, rather that one held within an organization. The questions in Exhibit 3.18 are intended for both participants and sponsors, although some questions may seem more appropriate for one group or the other.
80
c03.indd 80
Evaluating the Impact of Leadership Development
2/1/08 3:53:48 PM
Exhibit 3.17. End-of-Program Focus Group
FA C I L I TAT O R ’ S G U I D E D R A F T Introduction • Hello, I’m and I’ll facilitate our discussion today, which will take about an hour. • You have just attended the MNO leadership development program. I’d like to talk to you about the program and your reactions to it. • Let’s spend a couple of minutes talking about how groups like this work. Have any of you been in a focus group like this before? • My role during our group is to ask a lot of questions. I work for the . My job is to help the organization gather information about the program. I will then take that information and write a report summarizing the main themes in order to improve the program. • In order to ensure that all the information you provide to us is accurately captured, our group is going to be audiotaped. Please speak at least as loudly as I am speaking (stress this for accuracy of data). Also, please speak one at a time and avoid side conversations with your neighbors because I’ll miss what is being said. Your name will not be associated with any quotes. • We are not being videotaped and there will be no outside observers. • The most important thing in a group like this is to hear what everyone in the group thinks. But since we are on a very tight schedule, please don’t feel offended if I interrupt sometimes and move the conversation to another topic or another person. (Continued)
Designing and Conducting the Evaluation
c03.indd 81
81
2/1/08 3:53:49 PM
Reasons for Choosing This Program • How did you hear about the ? • What kind of things did you hear about the and its programs?
Content of Program • Did this program meet your needs? If yes, in what way? If no, what was missing? (Here’s the heart of it and where you should spend the bulk of your time.) I want to talk to you a little bit now on some specific parts of the program.
Leadership Challenge Preprogram Interviews • Before the program, you had an interview with several individuals in your organization, including your boss, to discuss your leadership challenge. Did you find this to be helpful in your preparation and understanding of the program? (Probe for why or why not.)
Role of the Leadership Challenge During the Program • How did knowing your challenge help you focus on achieving your goals during the program? • What role did your leadership challenge play in helping you tie together what you learned in the program and the setting of your developmental goals?
Value of Assessments (, , ) • How many of you have taken before? If yes, did you remember your type? If you have taken before, did we present any material that was new to you? • How well did we do in helping you to understand your results and how to apply them? (Continued)
82
c03.indd 82
Evaluating the Impact of Leadership Development
2/1/08 3:53:49 PM
• Had you ever been exposed to the before? Did you learn anything new as a result of the ? How well did we do in helping you to understand your results and how to apply them?
Simulation Let’s talk about Tuesday’s simulation for a moment. • What was most helpful? Least helpful? Why? What surprised you about the simulation? • What was realistic about the simulation? • How would you describe the quality and benefit of the Memo Sets? • How would you describe the quality and benefit of the Information Center? • Ease of use • Quality of information provided • Overall value added to the simulation • Changes you would recommend
Consolidation of Learning • What are your thoughts on the process you went through yesterday afternoon to consolidate your learning? How about the one-on-one discussion with the trainer to clarify your feedback? • Were you able to make connections between the feedback you received from the simulation and your feedback from the three assessment instruments? If yes, what helped you make those connections? If no, can you describe the disconnect you experienced? Do you have recommendations to improve the connections? • What did you like about the process and what would you like to have seen presented differently? (Continued)
Designing and Conducting the Evaluation
c03.indd 83
83
2/1/08 3:53:49 PM
Technology • Now I’d like for us to talk about the hand-held PDAs that were used to take the questionnaire. Was there anything about using the PDAs that was confusing or difficult to understand? • Was there anything about the questionnaire you would like to see done differently?
Developmental Planning • What could be done to improve the developmental planning and goal-setting process?
Other • What was your most important or meaningful learning moment? • When you get back home and talk to your colleagues and friends about MNO, what is the one thing you will want them to know about the program and your experience in it? • If you could talk to the program manager of MNO, what would be the one thing you would tell her about this program? • Is there anything else I should know that you have not already shared?
Wrap-Up • Your feedback was very helpful and will help the improve the MNO program. As a small token of our appreciation for your participation, please select one of the publications on the table outside of the door. Thank you for coming.
84
c03.indd 84
Evaluating the Impact of Leadership Development
2/1/08 3:53:50 PM
Exhibit 3.18. Focus Group (Several Months After the Program) Ask respondents to be prepared to respond to the questions from their own perspectives.
DRAFT LIST OF QUESTIONS 1. Since the cohort last met, what can you add to your “story” of the
impact of your participation in the Leadership Academy on:
a. Your own leadership capacity b. The effectiveness of the people with whom you work c. The effectiveness of your organization
2. What outcomes has your action learning project delivered to date? What, if any, additional outcomes do you anticipate delivering?
3. How effective was your action learning project from a learning perspective?
4. What grade (A, B, C, D, F) would you give to the Leadership Academy as a whole, if it were subjected to a report card? Why?
5. Has being a member of the Leadership Academy increased the visibility of participants, including being considered for open positions? Can you provide an example?
(Continued)
Designing and Conducting the Evaluation
c03.indd 85
85
2/1/08 3:53:50 PM
6. What has been the effect of the Leadership Academy on your commitment to ABC? Are you more committed, less committed, or is your commitment about the same as compared to your commitment to ABC before you participated in the Leadership Academy?
7. As you look back over the Leadership Academy experience, how would you describe the support you’ve had (either as a participant or a sponsor)?
8. What has been missing, in terms of support you wish you’d had?
9. How would you evaluate the success of the Leadership Academy? Specifically, as a mechanism for a. Individual development b. Networking/building relationships c. Creating an impact on the organization d. Creating recognition of the Leadership Academy with CHP
10. What additional insights would you like to share with regard to the Leadership Academy?
Advice
• Be sure you are ready to facilitate a focus group before beginning. Focus groups are more than a process of asking questions. Group facilitation is extremely important to keeping the focus on the questions at hand rather than detouring into a tangential conversation.
86
c03.indd 86
Evaluating the Impact of Leadership Development
2/1/08 3:53:50 PM
• Be aware of whether all members of the focus group are sharing their perspectives. Do not let a few voices dominate. • Know the dynamics of the group—whether you will need to continually probe for details or the group members will openly communicate their perspectives. • Do not voice your opinions during the focus group; remain in the evaluator role. You may respond to factual questions that are asked by focus group participants. • Tell focus group participants the purpose of having a focus group, how data will be shared, and why. • When focus groups are used to obtain a quick response at the end of a program, ensure that you have allowed enough time to analyze the data and make a report or presentation quickly after the focus group takes place. In general, aggregate the data rather than reporting who said what. However, if there are examples that connect to a specific individual and those examples would be particularly powerful for the evaluation report, gain permission from the participant(s) before including those examples in the report. Dialogue Group dialogue is a special kind of conversation in which people listen intently for underlying meanings and assumptions. Dialogue requires that participants suspend their assumptions in a way that enables them to hear others’ perspectives objectively. This technique allows an open, creative conversation to take place, often freeing the participants in the conversation to become aware of and better understand different perspectives. Unlike a focus group, a dialogue allows the evaluator to remain relatively invisible (at his or her choosing) and to let the conversation take place between group members (McGuire & Palus, 2003). The evaluative results of a dialogue session come from participants talking about their experiences in a supportive atmosphere. To achieve this goal, share the ground rules in Exhibit 3.19 at the beginning of the session and ask all participants to agree to follow them. The value in this method is that it allows team members and groups of participants to interact with one another, exploring their perspectives and insights more deeply than they might in a focus group. You can use dialogue in your evaluation to understand the different perspectives of stakeholder groups and to clarify what the collected data mean. Designing and Conducting the Evaluation
c03.indd 87
87
2/1/08 3:53:50 PM
A group dialogue often requires little intervention by the evaluator. However, it may be necessary during the dialogue for evaluators to refocus the conversation on the topic at hand. Exhibit 3.20 shows some sample dialogue questions. Each question is followed by a description of the type of information that it potentially makes available (satisfaction, learning, and/or behavioral change). During the design of your own evaluation, you may want to use questions more specific to your situation and the organizational context. To gain the broadest possible perspective, balance the types of information you seek through dialogue.
Exhibit 3.19. Ground Rules for the Dialogue Method • Everybody has an equal place in the conversation. (It helps to sit in a circle if possible, without tables in the middle.) • Each person says only what he or she feels comfortable saying. • The group should agree on how the results of the dialogue will be used. • The group should protect the confidentiality of individual remarks. • Participants should speak to the middle of the circle and avoid one-on-one or side conversations. • There should be a balance between advocacy (statements of belief) and inquiry (questions of clarification). • Listen. Silence is o.k. Support any quiet or solitary voices. • Seek, surface, and explore multiple perspectives. • Participants should work together to collaboratively understand shared experiences. • Don’t try to solve problems during the dialogue session. • Spend time at the end for debriefing. What was good about the conversation? What was frustrating? What was learned that could be used to carry the process forward?
88
c03.indd 88
Evaluating the Impact of Leadership Development. Copyright © 2008 by Center for Creative Leadership. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
2/1/08 3:53:51 PM
Advice
• See ground rules (Exhibit 3.19). • When the dialogue method is used to get a quick response at the end of a program, ensure that you have allowed enough time to analyze the data and make a report or presentation quickly after the dialogue takes place. • In general, aggregate the data rather than reporting who said what. However, if there are examples that connect to a specific individual and those examples
Exhibit 3.20. Group Dialogue Questions (Example) Dialogue Question
Potential Information
What stands out for you when
Satisfaction, learning
you think about the experience of the initiative itself? What stands out for you when
Learning, behavioral change
you think about the impacts of the initiative? What can you do now that you couldn’t or didn’t do
Satisfaction, learning, behavioral change
before the initiative? What’s the value of the
Behavioral change
initiative experience? What do you see as the
Satisfaction
impact on other people? On the team as a whole? What could be done
Satisfaction
differently to improve the initiative? Are we missing anything in our dialogue so far?
Designing and Conducting the Evaluation
c03.indd 89
89
2/1/08 3:53:51 PM
would be particularly powerful for the evaluation report, gain permission from the participant(s) before including those examples in the report. • It is useful to report notable shifts in the dialogue. Because participants can take the dialogue where they want to, it is important to note when the group touched on a topic quickly and then moved to another topic, where the group stayed for a significant amount of time, when there was more or less emotion about a given topic, and other significant aspects of the dialogue that stand out to you.
Workplace or Community Statistics Workplace statistics include information that organizations often gather on absenteeism, promotions, communication breakdowns, and grievances. Communities may collect statistics on events, issues, and the like. Evaluators can obtain that information from the organization or community and analyze it in relation to the leadership development initiative and its objectives. Before requesting and using statistics in an evaluation project, carefully determine which statistics are likely to change as a result of an individual’s, team’s, or group’s participation in the initiative. Statistics are useful as an evaluation method when a development initiative is designed to meet specific outcomes. For example, when participants are expected (as part of their action plan based on their development experience) to work differently with at-risk employees to prevent situations that might lead to grievances, it is appropriate and useful to examine the change in frequency and severity of grievances that employees file. When the head of a community health department, as a result of participating in leadership development, sets goals related to encouraging better community relations, it is useful to examine health department data related to wait times and/or return visits by the public. Statistics aren’t useful if there aren’t links between the statistics, the expected outcomes, and the development initiative. (Refer to Chapter 2 to assess whether this method is an appropriate evaluation technique for your situation.) For examples of types of workplace statistics that may be appropriate to use in the evaluation of your leadership development initiative, please refer to Phillips (2000). The survey in Exhibit 3.21 was developed as a means of assessing self- and manager-reported evidence of impact resulting from a leadership development initiative. These statistics can also be collected directly from existing organizational databases or sources.
90
c03.indd 90
Evaluating the Impact of Leadership Development
2/1/08 3:53:52 PM
Exhibit 3.21. Workplace Statistics Survey (Example)
BUSINESS OUTCOMES Please provide a numeric rating for each of the outcomes listed below or indicate NR if the outcome is not relevant to your situation. My own development and improved leadership effectiveness have contributed to organizational change as expressed below: 1 Decreased dramatically 2 Decreased 3 Not changed 4 Increased 5 Increased dramatically NR Not relevant 1. Productivity within the group I lead has . . . 1
2
3
4
5
NR
2. Profit within the group I lead has . . . 1
2
3
4
5
NR
3. Level of trust and collaboration within the group I lead has . . . 1
2
3
4
5
NR
4. Customer satisfaction with the group I lead has . . . 1
2
3
4
5
NR
5. Customer loyalty to the group I lead has . . . 1
2
3
4
5
NR
6. Employee job satisfaction among my subordinate group has . . . 1
2
3
4
5
NR
7. Employee promotions among my subordinate group have . . . 1
2
3
4
5
NR
8. Employee job effectiveness among my subordinate group has . . . 1
2
3
4
5
NR (Continued)
Designing and Conducting the Evaluation
c03.indd 91
91
2/1/08 3:53:52 PM
9. Innovation within the group I lead has . . . 1
2
3
4
5
NR
10. Implementation of new ideas within the group I lead has . . . 1
2
3
4
5
NR
11. Successful completion of projects within the group I lead has . . . 1
2
3
4
5
NR
12. New products and services developed within the group I lead have . . . 1
2
3
4
5
NR
Please note that in the preceding items, higher scores are preferred. In the items that follow, lower scores are preferred. 13. Absenteeism among my subordinate group has . . . 1
2
3
4
5
NR
14. Tardiness among my subordinate group has . . . 1
2
3
4
5
NR
15. Employee turnover among my subordinate group has . . . 1
2
3
4
5
NR
16. Requests for employee transfers among my subordinate group have . . . 1 2 3 4 5 NR Provide quantitative evidence for the two areas (above) that have shown the most impact (for example, the percent of increased employee promotions among your subordinate group or money saved because of a particular change you’ve implemented).
Advice
• Statistics can be very useful, if the data represent direct or indirect outcomes of the leadership development program. Work with stakeholders to identify critical, logical, and realistic links. • Ensure that stakeholders understand what statistics will be part of the evaluation and their relationship to the program so that data provided to you will support your use of the data.
92
c03.indd 92
Evaluating the Impact of Leadership Development
2/1/08 3:53:52 PM
• Design the evaluation plan so that you are collecting data that can represent trends for a period of time that makes sense, relative to the changes you anticipate. • Understand any limitations on the reporting of statistics before you report them. When data represent human resources and/or confidential issues, you have to understand any agreements under which they were collected before you share them. • Report data in a way that is appropriately cognizant of the amount of time in which changes can realistically be expected.
Archival Documents and Data Documents that may provide helpful evaluation data include letters, journals, logs, position papers, notes, speeches, newspaper articles and editorials, annual reports, newsletters, case studies, evaluation and consultants’ reports, and photographs. Expenditure records; expense account vouchers; financial information; performance records; professional, business, or school directories; student and participant achievement and performance test records; state and federal regulatory records; attendance records; competency and attitude score records; and electronic mail records may also be helpful. These data can be helpful in understanding the history or background of a program or situation; in tracking the number of times that something has occurred; in helping develop survey or interview questions or areas on which to focus an observation; in identifying patterns of participation, interest in, or attitudes toward a program; and in better understanding an issue that people are unable or unwilling to talk about. Document analysis has several advantages as a method of data collection. Documents and records are often plentiful (stored usually in the organization’s archives) and inexpensive to collect. Data from documents and records may provide useful chronological detail (possibly more accurately than a chronology built from interviews). The data gathered from documents and records are less subject to an evaluator’s bias than data from interviews and observation. When used with other methods, documents and records provide a contextual richness for interpreting other data collected during the evaluation. There are also some limitations. For example, samples of documents or records may not be representative (notes from some meetings may exist, but not from
Designing and Conducting the Evaluation
c03.indd 93
93
2/1/08 3:53:53 PM
others). Personal documents may reflect a person’s bias. Deliberate deception or manipulation of information is possible if the document’s writer had a desire to express a certain viewpoint. Also, records may contain incomplete or out-of-date information. See Exhibit 3.22 for an example related to a school system involved in a multiyear, multi-school leadership development initiative. Because the school system’s budget was so influential on its operations and climate, it was a factor contributing Exhibit 3.22. Document Schools May Need $4 Million More mm-dd-yy By ,<Staff Writer> Schools will need at least another $4 million next year to cover increases in basic costs, <Superintendent name> said <Monday>. <Superintendent name> said the school system will need an additional $4 million to $5 million to cover increases in three main areas: the district’s share of state-mandated pay and benefit increases for teachers and other employees, an anticipated rise in utility costs, and student enrollment and costs associated with cleaning and air conditioning new classroom additions. A second straight pay raise for teachers would take even more money, <superintendent name> said, as would expanding academic programs or adding new ones. <Superintendent name> hinted that he might suggest cutting jobs to find money for these areas. <Superintendent name> declined to be more specific. He said he will present a more detailed plan to the Board of Education sometime next month. “There’s nothing we’re going to recommend that will be popular. There’s nothing we’re going to recommend that will be easy,” <superintendent name> said. “But the board told us to come up with a plan.” <Superintendent name> ’s comments came during the first meeting this year of the School Budget Committee, a ten-person group that (Continued)
94
c03.indd 94
Evaluating the Impact of Leadership Development
2/1/08 3:53:53 PM
includes school board members, county commissioners, and the top executive and budget officer from the county and the school district. County Manager and budget director , who both attended Monday’s committee meeting, said it is too early to say whether the county will have more revenue next year to cover increased school costs. “We’re months away from that,” said. The county commissioners increased the school district’s budget last year by $7.6 million, or 7 percent. The commissioners paid for the increase by raising the property tax rate by about 6 percent. The county supplies about 30 percent of the school district’s annual budget. Most of the rest comes from the state. School district officials are already taking a look at their budget request for the school year. School officials said they are bracing for state-mandated increases in salaries, employee retirement, and hospitalization insurance. Energy costs and student enrollment also are expected to increase next year. The district also is adding 767,000 square feet of new classroom space. Another 1.4 million square feet will be air conditioned for the first time. To increase teacher pay or add academic programs, the school district will have to get more money from the county or dig it out of its existing budget. Last year, the system reallocated $4 million to lower class sizes at fifteen elementary schools and expand summer school. That money is no longer there, <superintendent name> said. The district raised money last year for its class-size reduction plan by cutting ninety-four teacher assistant jobs. <Superintendent name> hinted Monday that he might consider cutting more of these jobs. <Superintendent name> also said he is looking at another potentially unpopular step: cutting the amount of planning time that middle school teachers have during the school day. This step would increase the time that teachers spend teaching, which means middle schools will need fewer teachers. Staff writer contributed to this report.
Designing and Conducting the Evaluation
c03.indd 95
95
2/1/08 3:53:53 PM
to the effectiveness of the leadership development initiative and was therefore tracked. Advice
• In many evaluation situations, the amount of archival data that could possibly be collected is almost endless. Be sure that you collect only what is needed and realistic to report. • If at all possible, make good use of archival data early in an evaluation process to help in creating other methods of data collection for later use. • When leadership development programs include projects, action learning experiences, or other forms of action that participants are expected to report on, the presentations and reports from those projects are excellent documents to include in your evaluation, as they address actions and outcomes directly tied to the program itself. • Use examples from the archival data when possible, gaining permission before reporting anything that might be confidential. • If it is important to make samples available to key stakeholders who receive your report, consider posting the samples in an electronic format accessible to recipients.
DATA STORAGE AND TREATMENT Data collected as part of an evaluation often contain private and confidential information. And there is also often a lot of data, so having a clear storage and treatment plan is helpful. Here are some things to keep in mind: • Store confidential data in a secure and private area (a locked room or cabinet, a password-protected electronic file, etc.). If you are not certain about how private or confidential the data are, it is better to be overly cautious. • Keep a copy of data that are critical to an evaluation. To save space, you may want to scan paper documents and keep them stored electronically. • For confidential data, consider using a coding scheme (an identification number, for example) rather than a person’s or group’s name. If data are compromised, no one will know for sure whose data they are. • If you combine information from different sources, make sure you have a common identifier for each data source. For instance, use the same identification
96
c03.indd 96
Evaluating the Impact of Leadership Development
2/1/08 3:53:54 PM
code for a person’s interview and survey data. This approach also helps eliminate the need to gather basic information multiple times. You can link information, reducing redundancy and people’s frustration with providing the same information over and over. A response rate is the percentage of individuals invited to participate in a datacollection activity who actually complete it. If you send a survey to one hundred people and thirty people return a completed survey, your response rate is 30 percent. In some evaluations you may identify a sample, or subgroup, of people—program participants, for example—to respond to a survey. If there are five hundred program participants and you decide to randomly select one hundred for your survey and fifty respond, you have a 50 percent response rate, even though you receive completed surveys from 10 percent of all participants. Getting people to respond to a survey or any other request for information can be difficult, especially when they don’t have to. Some tips for increasing the likelihood that people will respond are listed in Exhibit 3.23. A final checklist is provided in Exhibit 3.24.
Exhibit 3.23. Increasing Response Rates Let people know a request is coming. If you know you will be gathering data from program participants, for instance, let them know during the program that they will be contacted after the program. Explain why participating in the process is important, how information will be used, and whether there are benefits to them from participating in the process. Taking the time during the program to explain things and answer questions can really pay off. Personalize your request to participate. Send your request to the person, not the role (for example, “Dear Kelly” rather than “Dear Program Participant”). It can also be helpful to have the request come from someone known by the people you want to participate (for example, the president of the foundation or a program trainer, rather than an evaluator they have never met or heard of). (Continued )
Designing and Conducting the Evaluation
c03.indd 97
97
2/1/08 3:53:54 PM
Clearly state the purpose and benefit. In short, let them know how you will use the information and why their perspectives are important. Keep your invitation short and simple. Tell them how long the survey will take, how the information will be used, and with whom data will be shared. Then thank them for their time. Remind people to complete the process. It is important to remind people to complete data collection, but not to hassle them. Usually two reminder e-mail messages will do the trick. Whenever possible, remind only the people who have not yet completed the process. Consider offering incentives. These incentives can be small tokens, gift certificates, a day off, etc. If funds are limited, you can use a raffle approach in which those completing the survey are entered into a drawing for a prize. If you don’t have the money for a prize, you may want to contact local merchants to see if they will donate one. In some cases the “prize” can be a donation to a cause people support. For example, “We will donate $1 to Charity X for every completed survey” or to try to get a minimum response: “If one hundred people complete the survey, we will donate $500 to Charity X.” Offer to share your results with participants. People who respond may want to see the final results. Offering to send them the final results, if appropriate, can boost participation. Results do not have to be printed; a link to a file online can save production and postage costs, not to mention being better for the environment. If reading results isn’t likely to encourage folks, you could host a community meeting to share insights or a special presentation (in person or online) to share information and perhaps answer questions.
98
c03.indd 98
Evaluating the Impact of Leadership Development
2/1/08 3:53:55 PM
Exhibit 3.24. Designing and Conducting the Evaluation Checklist ❏ Examine impact from multiple perspectives. ❏ Assess the different kinds of change that can be observed. ❏ Use multiple data-collection methods. ❏ Look at change over time. ❏ Assess individual- and group-level change. ❏ Use control groups for comparison. ❏ Use time-series designs for comparison. ❏ If you have a large enough sample, conduct a pilot study. ❏ Consider various methods of measuring change. ❏ Be mindful of issues regarding the measurement of change. ❏ Design the evaluation plan, choosing appropriate evaluation methods. ❏ Plan for data storage and treatment.
Designing and Conducting the Evaluation
c03.indd 99
99
2/1/08 3:53:55 PM
Communicating and Using Evaluation Findings
0
chapter FOUR
E
valuations often fail to live up to their full promise because what is learned is not broadly communicated and used. Even when evaluation findings are used, they frequently are only used to a fraction of their potential. It’s important that evaluators, participants, organizations, funders, and communities understand that the evaluation process doesn’t end when all of the data have been collected, analyzed, and interpreted. On the contrary, that is the time to revisit the reasons the evaluation was originally commissioned. To ensure that your evaluation findings will be used to support learning, make decisions, and/or make improvements, you need to complete four critical activities: effectively communicate the results, identify specific courses of action, develop a detailed action plan, and monitor the action plan’s implementation.
These four activities create a greater likelihood that your evaluation’s recommendations will be carefully considered and translated into action and that the actions taken based on the evaluation findings will be realistic and sensitive to the organization’s culture and context. They will also help the organization identify and manage any potential barriers or obstacles to change. These activities allow those individuals affected by the changes to become involved in planning the changes. Perhaps most important, these activities lay the groundwork for the organization’s
101
c04.indd 101
2/1/08 3:54:27 PM
using the evaluation findings as part of a continuous process of improvement and learning (Preskill & Torres, 1999).
EFFECTIVELY COMMUNICATE RESULTS Most evaluations result in a final report that is submitted to the primary stakeholder, client, or foundation. All too often, only a few people read that final report, so the degree of learning and change that occurs as a result of the evaluation is limited. Evaluators can counter this tendency by carefully considering their reporting strategy while keeping the goal of a usable evaluation in mind. Organizations and foundations will more often use evaluation reports that are credible, relevant (to stakeholders’ interests, expectations, and communication preferences), accurate, and fair. In order to effectively communicate, you will need to consider how you will distribute what information to whom. Communication Mechanisms One tactic to consider is the use of multiple methods for communicating and reporting the process of the evaluation and its results. A broader communication strategy can help to more widely distribute the lessons arising from development initiatives, and it also emphasizes the idea that the results of an evaluation provide information that can and should be used in an organizational learning process, that the evaluation results are not the final goal—learning is. Informing a variety of audiences about the findings of the evaluation and how the results will be used increases the credibility of the evaluation effort, communicates to stakeholders that they were heard and that the results will be acted on, and prepares stakeholders for future evaluations by demonstrating their potential value. Communication channels for evaluation information can include the following: • Oral and multimedia presentations • Executive summaries and flyers • Memos • Case studies • Postcards • E-mail and voice mail messages
102
c04.indd 102
Evaluating the Impact of Leadership Development
2/1/08 3:54:28 PM
• Photo essays, scenarios • Question-and-answer sessions • Web pages • Electronic documents • Newsletters • Bulletins • Posters Distributing interim reports before the release of a final report is another useful and effective communication tactic. Doing so will enable stakeholders to make critical changes to the initiative while it is still under way and it is still possible to effect change. In addition, high-level leaders and executives who may not have time for or interest in a complete and detailed account of findings may benefit from an interim overview of results and information about the specific questions of interest to them—information that helps them make necessary changes or decisions. Finally, when a development initiative is designed to occur in stages or has separate components, evaluation reports can be issued at each stage as well as rolled up in a final, comprehensive report. See Exhibits 4.1 and 4.2. Who Receives Information? Take into account the stakeholder group’s background, communication preferences, and information needs (much of this information should be available from the initial planning phase of the evaluation). If the information is not available, ask for it. (What information do stakeholder groups want? When do they need or want the information? How do they best receive information?) The most articulate evaluation report is not worth much if the key stakeholders do not read it or do not see it as relevant. What Information and When? Beyond choosing a variety of media and understanding which people you need to communicate with, you also need to understand the information needs of different groups. You can use that information to tailor the timing, content, and communication channel to the information you are reporting and to the stakeholder groups receiving it. Not everyone will be interested in or have the time to think about all the information an evaluation will glean. Strategically thinking about your communication approach will improve the likelihood that people are getting the information they need and can use.
Communicating and Using Evaluation Findings
c04.indd 103
103
2/1/08 3:54:28 PM
Exhibit 4.1. Basic Elements of a Final Evaluation Report A final evaluation report should be clearly written and to the point. Avoid words that may be unfamiliar to the intended audience. If you must use technical language or jargon, explain it in layperson terms. If appropriate, use graphs or narratives to illustrate a point. The report should be attractive and professional. The report’s presentation should not be overwhelming; use headings and page numbers to help orient the reader. When appropriately used, color can add interest and clarity to the report. Do not skip over the most basic elements of an effective report: correct grammar, spelling, and punctuation. For organizing purposes, the following list can be a helpful guide: Executive summary. Highlight the most important parts of the findings, including recommendations. Purpose of the evaluation. Indicate why the evaluation was conducted, including the core evaluation questions and stakeholders. Description of initiative and sample. Include the title of the initiative, what the initiative sought to accomplish, and who participated in it. Basic demographic information about program participants can be helpful, as well as a description of the stakeholders from whom data were collected for the evaluation. Caveats about the evaluation. Usually evaluators are not able to gather data from all stakeholder groups or all people within a stakeholder group. It is important to acknowledge what information or perspectives might be missing. Overview of processes used and analysis. What data were collected, how, and when? Was there anything out of the ordinary about the methodology and/or processes used? Provide samples of your evaluation methodologies as appendices. Summary of data (with multiple subsections). The section could be divided by evaluation questions, by results related to the (Continued)
104
c04.indd 104
Evaluating the Impact of Leadership Development
2/1/08 3:54:29 PM
implementation of the program versus the outcomes (short-, mid-, and long-term), or by overall lessons learned. The organization of sections depends very much on what was examined by the evaluation, what was found, and what would be the most salient way to present it to your stakeholders. Conclusions and recommendations. This section should be extremely clear so that your stakeholders know exactly what has occurred and what changes they may need to make. Appendices (supporting materials). Generally, include a copy of datacollection tools, data-analysis tables (detailed), and other resources that are pertinent to understanding the report.
Exhibit 4.2. Advice for Reporting Data Quantitative Data • If you are sharing only a few numbers, you may want to include them in the text rather than as a separate table. • Depending on your audience, you may want to include graphic representations (such as pie and bar graphs) in the text of the report to highlight important information and provide detailed tables of all the information in the appendices. • Let readers know what to look for or pay attention to in the tables and graphs. It may be self-evident to you that a rating of 4.8 on a scale of 1 to 5 (with 5 ⫽ excellent) is positive, but that might not be clear to readers. • You probably do not want to include a detailed discussion of every data point, just the data most critical for readers of the report to understand. (Continued)
Communicating and Using Evaluation Findings
c04.indd 105
105
2/1/08 3:54:29 PM
• Include the sample size and a descriptive title on all tables and graphs. • Explain any abbreviations used, as well as the numbers associated with response options. For example, what does a rating of 1 indicate? • Describe how data were collected and analyzed in the text of the report. • Use the numbers that reflect the level of precision needed. For example, don’t report 67.987 percent when 68 percent would be adequate. • Provide basic explanations of sophisticated statistics and provide information to help readers interpret results. • Just reporting means or averages can sometimes be misleading, as they do not indicate the variability of responses, so you may want to focus on sharing the frequencies associated with the various response options. • Explain the connection between the data and any recommendations you make. For instance, “Based on the user satisfaction ratings, we recommend the following . . .”
Qualitative Data • Report the themes from the data. Include definitions with illustrative quotes or examples to provide richness. • If themes and quotes conflict with one another, note the discrepancy. If there appear to be reasons for it, provide an explanation. • It is helpful to indicate the number of comments (or the number of individuals making comments) grouped into each theme to give a sense of how prevalent a theme is. • In some cases, you may want to include a graph to illustrate the number of times a theme is mentioned or the number of individuals who mention a theme. This can be as simple as the number of positive and negative comments about a program component. (Continued)
106
c04.indd 106
Evaluating the Impact of Leadership Development
2/1/08 3:54:30 PM
• If you have promised individuals confidentiality and are using illustrative quotes or examples, be sure to remove names, titles, and any other information that could be used to identify who provided the information. It may be difficult to determine what information could be used to identify individuals—either the speaker or the target of the response—but you need to do whatever is necessary to guarantee confidentiality if it was promised.
IDENTIFY SPECIFIC COURSES OF ACTION The most useful evaluation reports provide both critical findings and recommended actions. The actions must be specific and must follow logically from the evaluation results. A good process to use in identifying specific recommendations involves four steps: 1.
Based on the evaluation’s results, determine what changes are needed in what areas. Broadly speaking, those changes can be to the support structure in the organization or community, and/or to the initiative itself.
2.
Examine possible revisions to the initiative and to specific areas that strengthen appropriate support structures.
3.
Explore the need to follow up the leadership development initiative with another activity.
4.
Gather and consult with the stakeholders. What changes are they interested in making? What are their priorities?
Based on the Evaluation’s Results, Determine What Changes Are Needed in What Areas One possible outcome of a leadership development initiative is that it has the impact stakeholders and participants have expected and hoped for. In that case, no remedial action is necessary, but the organization or foundation might want to consider its next step in the development process for its employees or grantees. Stakeholders may want to share and celebrate the success of the initiative. Another possible outcome is that the data show the initiative needs to be revised to achieve maximum impact. In this case, the specific areas and suggestions for
Communicating and Using Evaluation Findings
c04.indd 107
107
2/1/08 3:54:30 PM
revision, based on the evaluation findings, should be identified and prioritized by relevant stakeholder groups. Still another outcome might be that participants gain a number of new insights but aren’t able to employ much of what they learn because of systemic obstacles in their work environment or a lack of understanding about how to transfer what they’ve learned back into their own situations. Evaluators can help identify barriers, but it is primarily the responsibility of the organization, community, or funder to address those issues. Depending on the nature of the obstacles, subsequent training or other development interventions may be needed. For example, if the leadership development initiative involved a culture or climate survey, the information gathered from the survey may indicate the work the organization or grantee needs to do. An organization may want to update reward systems, policies, or communication processes to better reflect the vision of the organization. A foundation may also have to develop processes to communicate its vision or expectations to grantees. A review and revision of a leadership development agenda may also be needed. Perhaps the leadership competencies being developed don’t align with a business strategy, a community’s values, or a foundation’s culture. It’s best if stakeholders examine these issues before the leadership development initiative, but sometimes that’s not possible. Changes in other contextual factors may occur during the initiative or may not be readily apparent before the initiative’s start. It’s important to maintain attention on issues of alignment throughout the initiative to best ensure its success. Other experts, such as organizational development professionals, may have to be consulted if there is a substantial incongruity among systems, processes, outcomes, and other factors. Examine Possible Revisions to the Initiative and to Specific Areas That Strengthen Support Structures It’s possible that an evaluation’s findings may indicate a need for minor or major revision to the development initiative, perhaps including additional learning needs for individuals or groups and improving specific elements to support learning more broadly, thereby increasing an intervention’s appeal or effect. Based on the evaluation findings, a group of stakeholders may assume the task of redesigning aspects of the initiative.
108
c04.indd 108
Evaluating the Impact of Leadership Development
2/1/08 3:54:30 PM
Explore the Need to Follow Up the Leadership Development Initiative with Another Activity After the results of your evaluation are in, it’s a good time to explore with participants and other stakeholders the need for follow-up developmental activities. For example, coaching may be in order for some or all of them, and some participants may benefit more from additional specific skills-based training. When follow-up activities are included, the leadership development experience is extended and is more likely to be effective. Participants may also desire and benefit from additional support, such as meetings with mentors or coaches. Alumni events may also be helpful. Gather and Consult with the Stakeholders The most effective means for determining how to use what is learned from the evaluation is to bring together everyone with a stake in the evaluation and the initiative. This includes not just participants and stakeholders with an interest in the development initiative but also people who might be affected by changes made as a result of the findings and those individuals responsible for implementing the intended changes. The best time to do this is after you’ve reported evaluation results, which allows stakeholders some time to think about the results and correct any misunderstandings or errors in the report. Stakeholders should be encouraged to use data to respond to any perceived inaccuracies in the report because this creates a shared responsibility for report accuracy. If your evaluation results illustrate the shortcomings of a specific group in particular (such as the need for participants’ managers to become better development coaches for their direct reports), it may be wise to allow that group time to devise a course of action before bringing in all the stakeholders. The purpose in giving this time is to allow that specific group time to form a response and take ownership of the situation, not to put them in a defensive position.
DEVELOP A DETAILED ACTION PLAN Once key stakeholders have discussed potential actions, the next step for them is to develop an action plan. Evaluators may or may not be involved in or lead the actionplan process. An action plan is a tool for implementing the lessons that result from
Communicating and Using Evaluation Findings
c04.indd 109
109
2/1/08 3:54:31 PM
an evaluation study. Certainly, the lessons should point to areas of leadership development initiatives that are in need of revision, and in addition, they can indicate areas (such as systems and processes) that aren’t supporting those initiatives. Not all stakeholders have to be involved in the development of the action plan, although it’s often helpful to have as much involvement as possible. You can determine who needs to be involved by examining where action is needed and who has the responsibility and authority to enact change in that area. The action plan should outline specific tasks, identify individuals and groups who are responsible for carrying them out, include necessary resources for implementing specified actions, and include a timeline for carrying them out. As a guide to creating an action-plan document, consider or recommend these questions: • What must happen and why? What are the specific actions to be taken? How are the actions indicated by the evaluation results? • Who will make it happen? What group or individual is responsible for implementing the proposed action? • Who else has to be involved? What additional groups or individuals are required (for example, to participate in the action, to provide approval for it, or to play a support role)? • What barriers are anticipated? What obstacles to implementation exist? How will obstacles be addressed? • In what order does it have to happen? Do the steps have to occur in a particular order? • When does it have to happen? Is there a deadline for initiating and completing each step of the action plan? • How will the stakeholders know it has happened? What indicators will be observed to determine whether each step of the action plan has been initiated and completed? • How will success be determined? What indicators will be observed to measure the success of each step in the action plan? The action plan that results from this effort must be distributed to all those involved with the proposed actions so that all are clear about their roles in implementing the plan.
110
c04.indd 110
Evaluating the Impact of Leadership Development
2/1/08 3:54:31 PM
MONITOR THE ACTION PLAN ’ S IMPLEMENTATION As an action plan is implemented, its progress must be monitored and communicated. An evaluator may or may not be involved in this aspect of the work. Monitoring the action plan’s implementation means following up with individuals and groups responsible for specific actions, as indicated in the plan, to see how well they are progressing and what, if anything, has to be modified. One way to monitor the plan is to periodically convene the group that developed the action plan and have each member describe how well the actions are being implemented and what help is required to continue the process. If the members of the group are geographically dispersed, you can conduct these monitoring sessions using Internet capabilities, videoconferencing systems, shared websites, or e-mail. Regardless of the method used, all of those involved should become aware of the extent to which the plan is being implemented and what impact the changes are having on individuals, groups, and the organization. The group might even decide that, as a result of making certain changes, additional evaluation needs have surfaced.
USE EVALUATION TO CREATE BROADER CHANGE AND LEARNING Evaluation measures more than processes and impact. The process has the potential to create change in (and for) individuals, organizations, and communities. In addition to identifying gaps in, barriers to, and support for leadership development, evaluation can be used to help gather possible solutions and solicit ideas for improving a development effort. These changes can affect broad or small groups of people, or individuals, depending on the type of change and the effect it has at various levels. These changes also provide an opportunity for additional evaluation and learning. Important outcomes of evaluation include the individual and group learning that occurs as a result of participating in the evaluation and using the evaluation’s results. When individuals and groups reflect on their experiences and share what they’ve learned with others, the organization or community as a whole learns. The challenge, however, is in finding ways to capture and share this learning. One tactic is to develop a learning database that employees, volunteers, and/or other stakeholders can access on an as-needed basis. Another tactic is to create a discussion area online—on an intranet, for example, or via a blog. Yet another is
Communicating and Using Evaluation Findings
c04.indd 111
111
2/1/08 3:54:32 PM
Exhibit 4.3. Appropriate Use of Evaluation Data The collection, storage, and use of data representing individuals should adhere to appropriate and professional standards (Joint Committee on Standards for Educational Evaluation, 1994). Following these standards helps reduce ethical problems that can arise during the evaluation process. When you’re summarizing the purpose of the evaluation in your report, it’s a good idea to include a statement that any uses of the information not sanctioned by these guidelines may be misguided and erroneous because the data collected during the evaluation may not be relevant to other conclusions.
to publish, in internal or external newsletters and bulletins, the lessons revealed through participation in the development initiative and in the evaluation’s results. Each of these methods is a vehicle for distributing information more broadly so that people and groups can learn from each other. (For a list of resources related to learning through evaluation, see pages 117 through 128 at the end of this book.) In creating an evaluation plan that fosters learning, it’s important to consider guidelines that make the process less susceptible to misuse. (See Exhibit 4.3.) Three specific issues are especially relevant to the evaluation of leadership development initiatives: maintaining confidentiality of individual data; evaluating the initiative, not the individuals; and linking the evaluation to broader goals and strategies. Maintaining confidentiality of individual data builds evaluator credibility and trust that the process will be used to evaluate the initiative, not the individual. Individual data should not be shared with anyone other than that individual, unless that individual gives specific permission to share the data with other specific people. Individuals (and their observers) will often be more candid and honest in their responses when they know their data will not be shared with others except in aggregate form. Confidentiality can mean different things to different people, so clarify how information will be shared and with whom. Leadership development is intended for individual, team, organizational, community, and/or societal improvement. The evaluation data collected should be
112
c04.indd 112
Evaluating the Impact of Leadership Development
2/1/08 3:54:32 PM
used to enhance the initiative itself, to foster changes in participants’ leadership knowledge and skills, to encourage relevant changes in the organization and/or community, and to illustrate relevant factors related to the success of the initiative. Evaluators should encourage organizations not to use the collected data for performance appraisals or for administrative decisions regarding individual performance, since the collection of that kind of information is held to a different set of legal and professional standards (Joint Committee on Standards for Educational Evaluation, 1994). You can promote this value by aggregating any individual data collected during the evaluation to the group level. Evaluation should be part of strategic intent. It’s not intended to be a valueless exercise to which people are asked to give time, energy, and resources, but which is not used for productive purposes. Just as leadership development should be linked to strategy, so should evaluation be considered strategically important and expected to contribute to the achievement of goals and learning agendas. Evaluation is often a complex endeavor, and it’s impossible for any single book to provide all the information that may be needed for a particular situation. This brief book has provided a basic overview of the evaluative process. It has also described and provided examples of techniques, strategies, and guidelines that you can use to ensure that your evaluation is not an isolated process. By linking your evaluation to the design and results of development initiatives and to broader goals, you can help build systems and processes that augment the individual and group impact of leadership development.
Communicating and Using Evaluation Findings
c04.indd 113
113
2/1/08 3:54:32 PM
Exhibit 4.4. Communicating and Using Evaluation Findings: A Checklist ❑ Draft final evaluation report. ❑ Obtain key stakeholders’ feedback on and input to the final report. ❑ Disseminate report through different media. ❑ Determine what changes are needed in what areas. ❑ Examine possible revisions to the initiative and to specific areas that strengthen support structures. ❑ Explore the need to follow up the leadership development initiative with another activity. ❑ Gather and consult with the stakeholders. ❑ Suggest and possibly help develop a detailed action plan. ❑ Suggest and possibly help create a process for monitoring the action plan’s implementation. ❑ Suggest ways the evaluation can help create change and learning.
114
c04.indd 114
Evaluating the Impact of Leadership Development
2/1/08 3:54:33 PM
R E F E R E N C E S
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage. Fetterman, D. M., & Wandersman, A. (Eds.). (2004). Empowerment evaluation principles in practice. New York: The Guilford Press. Fitzpatrick, J. L., Worthen, B. R., & Sanders, J. R. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). New York: Longman. Greenbaum, T. L. (1999). Moderating focus groups: A practical guide for group facilitation. Thousand Oaks, CA: Sage. Hannum, K. M., Martineau, J. W., & Reinelt, C. (Eds.). (2007). The handbook of leadership development evaluation. San Francisco, CA: Jossey-Bass. Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, D. W., & Gerber, S. R. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3(1), 1–23. Inouye, T., Cao Yu, H., & Adefuin, J. (2005, January). Commissioning multicultural evaluation: A foundation resource guide. In partnership with Social Policy Research Associates. Oakland, CA: The California Endowment’s Diversity in Health Education Project. Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: How to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage. Kraemer, H. C., & Thiemann, S. (1987). How many subjects? Statistical power analysis in research. Thousand Oaks, CA: Sage. Martineau, J. W. (2004). Evaluating the impact of leader development. In C. D. McCauley & E. Van Velsor (Eds.), The Center for Creative Leadership handbook of leadership development (2nd ed., pp. 234–267). San Francisco, CA: Jossey-Bass. McCauley, C. D., Moxley, R. S., & Van Velsor, E. (Eds.). (1998). The Center for Creative Leadership handbook of leadership development (1st ed.). San Francisco, CA: Jossey-Bass.
115
bref.indd 115
2/1/08 4:34:47 PM
McGuire, J. B., & Palus, C. J. (2003). Conversation piece: Using dialogue as a tool for better leadership. Leadership in Action, 23(1), 8–11. Morgan, D. L. (1993). Successful focus groups: Advancing the state of the art. Thousand Oaks, CA: Sage. Morgan, D. L., & Krueger, R. A. (1997). The focus group kit (Vols. 1–6). Thousand Oaks, CA: Sage. Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), 311–320. Peterson, D. B. (1993). Measuring change: A psychometric approach to evaluating individual training outcomes. Paper presented at the eighth annual conference of the Society for Industrial and Organizational Psychology, San Francisco. Phillips, J. J. (2000). The consultant’s scorecard. New York: McGraw-Hill. Pratt, C., McGuigan, W., & Katzev, A. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341–349. Preskill, H., & Russ-Eft, D. (2004). Building evaluation capacity: 72 activities for teaching and training. Thousand Oaks, CA: Sage. Preskill, H., & Torres, R. T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage. Rohs, F. R. (1999). Response shift bias: A problem in evaluating leadership development with self-report pretest-posttest measures. Journal of Agricultural Education, 40(4), 28–37. Rohs, F. R., & Langone, C. A. (1997). Increased accuracy in measuring leadership impacts. Journal of Leadership Studies, 4(1), 150–158. Sawyer, D. (1987). Analyzing pseudo–time series interventions. Evaluation Review, 11(2), 178–196.
116
bref.indd 116
Evaluating the Impact of Leadership Development
2/1/08 4:34:47 PM
R E C O M M E N D E D
R E S O U R C E S
T
his collection of resources begins with a list of publications that can often be found online or at your local public, corporate, or university library.
The second group of resources consists of e-mail and discussion groups, or listservs, for those readers who intend to join the evaluation community or would like to have access to information from other evaluators. Some listservs have searchable archives within which users can locate additional information based on their specific needs. The third resource grouping is a list of professional organizations. These organizations often provide basic information about evaluation on their websites and sometimes offer channels for posting evaluation positions or proposals (for readers and users seeking evaluation services). Some of these organizations may have additional resources on their sites. Finally, the last resource group includes websites with links that provide access to resources that have a great deal of useful evaluation information. The reader is advised that, as with any web-based resource, the availability of this information can change, shift, and even disappear from the Internet without prior warning. All of the information listed here was current as of the date of this publication. Publications Abernathy, D. J. (1999). Thinking outside the evaluation box. Training & Development, 53(2), 18–23. Alkin, M. C. (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage.
117
bsource.indd 117
2/1/08 4:35:14 PM
Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta analysis of the relations among training criteria. Personnel Psychology, 50(2), 341–358. Bamberger, M., Rugh, J., & Mabry, L. (2005). RealWorld evaluation: Conducting evaluations with budget, time, data and political constraints. Thousand Oaks, CA: Sage. Barron, T. (2004). The link between leadership development and retention. Training & Development, 58(4), 58–65. Bassi, L. J., & Cheney, S. (1997). Benchmarking the best. Training & Development, 51(11), 60–64. Bassi, L. J., & McMurrer, D. P. (1998). Training investment can mean financial performance. Training & Development, 52(5), 40–43. Beywl, W., & Potter, P. (1998). RENOMO—A design tool for evaluations. Evaluation, 4(1), 53–71. Braverman, M. T., Constantine, N. A., & Slater, J. K. (Eds.). (2004). Foundations and evaluation: Contexts and practices for effective philanthropy. San Francisco, CA: Jossey-Bass. Bregman, P., & Jacobson, H. (2000). Yes, you can measure the business results of training. Training, 37(8), 69–72. Brinkerhof, R. O. (2003). The success case method: Find out quickly what’s working and what’s not. San Francisco, CA: Berrett-Koehler. Brown, S. M., & Seidner, C. J. (Eds.). (1998). Evaluating corporate training: Models and issues. Boston, MA: Kluwer Academic Publishers. Brunner, I., & Guzman, A. (1989). Participatory evaluation: A tool to assess projects and empower people. New Directions in Program Evaluation, 42, 9–17. Carter, L., Giber, D., & Goldsmith, M. (Eds.). (2001). Best practices in organization development and change: Culture, leadership, retention, performance, coaching. San Francisco, CA: Jossey-Bass. Catsambas, T. T., & Preskill, H. (2006). Reframing evaluation through appreciative inquiry. Thousand Oaks, CA: Sage. Chelimsky, E., & Shadish, W. R. (1997). Evaluation for the 21st century: A handbook. Thousand Oaks, CA: Sage. Chen, H. (1990). Theory-driven evaluations. Thousand Oaks, CA: Sage. Church, A. H., & Waclawski, J. (2001). Designing and using organizational surveys: A sevenstep process. San Francisco, CA: Jossey-Bass. Dainty, P., & Lucas, D. (1992). Clarifying the confusion: A practical framework for evaluating outdoor development programmes for managers. Management Education and Development, 23(2), 106–122. Daugherty, R. A., & Williams, S. U. (1997). The long-term impacts of leadership development: An assessment of a statewide program. Journal of Leadership Studies, 4(2), 101–115.
118
bsource.indd 118
Evaluating the Impact of Leadership Development
2/1/08 4:35:14 PM
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage. Denzin, N. K., & Lincoln, Y. S. (Eds.). (2003). Collecting and interpreting qualitative materials (2nd ed.). Thousand Oaks, CA: Sage. Dixon, N. M. (1987, August). Meet training’s goals without reaction forms. Personnel Journal, 66(8), 108–115. Earl, S., Carden, F., & Smutylo, T. (2002). Outcome mapping: Building learning and reflection into development programs. Ottawa, ON, Canada: International Development Resource Center. Edwards, J. E., Scott, J. C., & Nambury, S. R. (2003). The human resources programevaluation handbook. Thousand Oaks, CA: Sage. Ellis, K. (2005). What’s the ROI of ROI? Training, 42(1), 16–22. Festen, M., & Philbin, M. (2007). Level best: How small and grassroots nonprofits can tackle evaluation and talk results. San Francisco, CA: Jossey-Bass. Fetterman, D. M., & Wandersman, A. (Eds.). (2004). Empowerment evaluation principles in practice. New York: The Guilford Press. Fishel, B. (1998). A new perspective: How to get the real story from attitude surveys. Training, 35(2), 91–94. Fitz-Gibbon, C. T., & Morris, L. L. (1987). How to design a program evaluation. Thousand Oaks, CA: Sage. Fitzpatrick, J. L., Worthen, B. R., & Sanders, J. R. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). New York: Longman. Forss, K., Cracknell, B., & Samset, K. (1994). Can evaluation help an organization to learn? Evaluation Review, 18(5), 574–591. Frechtling, J. (2007). Logic modeling methods in program evaluation. San Francisco, CA: Jossey-Bass. Goldwasser, D. (2001). Beyond ROI. Training, 38(1), 82–90. Greenbaum, T. L. (1999). Moderating focus groups: A practical guide for group facilitation. Thousand Oaks, CA: Sage. Greene, J. G. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12(2), 91–116. Hackett, B. (1997, December). The value of training in the era of intellectual capital. The Conference Board (Report No. 1199-97-RR). Hannum, K. M., Martineau, J. W., & Reinelt, C. (Eds.). (2007). The handbook of leadership development evaluation. San Francisco, CA: Jossey-Bass. Herman, J. L., Morris, L. L., & Fitz-Gibbon, C. T. (1987). Evaluator’s handbook. Thousand Oaks, CA: Sage. Hodges, T. K. (2001). Linking learning and performance: A practical guide to measuring learning and on-the-job application. Boston, MA: Butterworth-Heinemann.
Recommended Resources
bsource.indd 119
119
2/1/08 4:35:15 PM
Hood, S., Hopson, R., & Frierson, H. (2005). The role of culture and cultural context in evaluation: A mandate for inclusion, the discovery of truth and understanding. Greenwich, CT: Information Age Press. Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, D. W., & Gerber, S. R. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3(1), 1–23. Inouye, T., Cao Yu, H., & Adefuin, J. (2005, January). Commissioning multicultural evaluation: A foundation resource guide. In partnership with Social Policy Research Associates. Oakland, CA: The California Endowment’s Diversity in Health Education Project. Insch, G. S., Moore, J. E., & Murphy, L. D. (1997). Content analysis in leadership research: Examples, procedures, and suggestions for future use. Leadership Quarterly, 8(1), 1–25. Jenlink, P. M. (1994). Using evaluation to understand the learning architecture of an organization. Evaluation and Program Planning, 17(3), 315–325. Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: How to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage. Julian, D. (1997). The utilization of the logic model as a system level planning and evaluation device. Evaluation and Program Planning, 20(3), 251–257. Kaplan, R. S., & Norton, D. P. (1996). The balanced scorecard. Boston, MA: Harvard Business School Press. King, J. A., Morris, L. L., & Fitz-Gibbon, C. T. (1987). How to assess program implementation. Thousand Oaks, CA: Sage. Kirkpatrick, D. L. (1998). Another look at evaluating training programs. Alexandria, VA: American Society for Training and Development. Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco, CA: Berrett-Koehler. Kraemer, H. C., & Thiemann, S. (1987). How many subjects? Statistical power analysis in research. Thousand Oaks, CA: Sage. Krueger, R. A. (1994). Focus groups: A practical guide for applied research. Thousand Oaks, CA: Sage. Kusek, J. C., & Rist, R. C. (2004). Ten steps to a results-based monitoring and evaluation system: A handbook for development practitioners. Washington, DC: World Bank Publications. Martineau, J. (1998). Using 360-degree surveys to assess change. In W. Tornow & M. London (Eds.), Maximizing the value of 360-degree feedback: A process for individual and organizational development (pp. 217–248). San Francisco, CA: Jossey-Bass. Martineau, J. W. (2004). Evaluating the impact of leader development. In C. D. McCauley & E. Van Velsor (Eds.), The Center for Creative Leadership handbook of leadership development (2nd ed., pp. 234–267). San Francisco, CA: Jossey-Bass.
120
bsource.indd 120
Evaluating the Impact of Leadership Development
2/1/08 4:35:15 PM
Mathison, S. (2004). Encyclopedia of evaluation. Thousand Oaks, CA: Sage. Mattessich, P. W. (2003). The manager’s guide to program evaluation: Planning, contracting, and managing for useful results. St. Paul, MN: Fieldstone Alliance. McCauley, C. D. (2006). Developmental assignments: Creating learning experiences without changing jobs. Greensboro, NC: Center for Creative Leadership. McCauley, C. D., Moxley, R. S., & Van Velsor, E. (Eds.). (1998). The Center for Creative Leadership handbook of leadership development (1st ed.). San Francisco, CA: Jossey-Bass. McCauley, C. D., & Van Velsor, E. (Eds.). (2004). The Center for Creative Leadership handbook of leadership development (2nd ed.). San Francisco, CA: Jossey-Bass. McGuire, J. B., & Palus, C. J. (2003). Conversation piece: Using dialogue as a tool for better leadership. Leadership in Action, 23(1), 8–11. Miner, N. J. (1998). Anonymous evaluations ain’t what they used to be! Training & Development, 52(3), 12–14. Mohr, L. B. (1995). Impact analysis for program evaluation. Thousand Oaks, CA: Sage. Monnier, E. (1997). Vertical partnerships: The opportunities and constraints which they pose for high quality evaluations. Evaluation, 3(1), 110–118. Morgan, D. L. (1993). Successful focus groups: Advancing the state of the art. Thousand Oaks, CA: Sage. Morgan, D. L., & Krueger, R. A. (1997). The focus group kit (Vols. 1–6). Thousand Oaks, CA: Sage. Morris, M. (Ed.). (2008). Evaluation ethics for best practice: Cases and commentaries. New York: The Guilford Press. Munck, M. E. (1997). Bridging the macro and micro levels in outcome evaluation. Evaluation, 3(3), 320–344. Murphy, J. R. (1997). Results first, change second. Training, 34(5), 58–67. O’Sullivan, R. G. (2004). Practicing evaluation: A collaborative approach. Thousand Oaks, CA: Sage. Owen, J. M., & Lambert, F. C. (1995). Roles for evaluation in learning organizations. Evaluation, 1(2), 237–250. Parry, S. B. (1997). Evaluating the impact of training: A collection of tools and techniques. Alexandria, VA: American Society for Training and Development. Patton, M. Q. (1987). How to use qualitative methods in evaluation. Thousand Oaks, CA: Sage. Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), 311–320. Patton, M. Q. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage. Peterson, D. B. (1993). Measuring change: A psychometric approach to evaluating individual training outcomes. Paper presented at the eighth annual conference of the Society for Industrial and Organizational Psychology, San Francisco. Peterson, R. A. (2000). Constructing effective questionnaires. Thousand Oaks, CA: Sage.
Recommended Resources
bsource.indd 121
121
2/1/08 4:35:16 PM
Phillips, J. J. (2000). The consultant’s scorecard. New York: McGraw-Hill. Phillips, J. J. (2003). Return on investment in training and performance improvement programs (2nd ed.). Boston, MA: Butterworth-Heinemann. Phillips, J. J., & Phillips, P. P. (2005). ROI at work: Best-practice case studies from the real world. Alexandria, VA: ASTD Press. Phillips, J. J., & Schmidt, L. (2004). The leadership scorecard. Boston, MA: Elsevier/ Butterworth-Heinemann. Phillips, P. P., & Phillips, J. J. (2005). Return on investment basics. Alexandria, VA: ASTD Press. Phillips, P. P., Phillips, J. J., Stone, R., & Burkett, H. (2006). The ROI field book. Boston, MA: Butterworth-Heinemann. Pratt, C., McGuigan, W., & Katzev, A. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341–349. Preskill, H. (1994). Evaluation’s role in enhancing organizational learning: A model for practice. Evaluation and Program Planning, 17(3), 291–297. Preskill, H., & Russ-Eft, D. (2004). Building evaluation capacity: 72 activities for teaching and training. Thousand Oaks, CA: Sage. Preskill, H., & Torres, R. T. (1999). Building capacity for organizational learning through evaluative inquiry. Evaluation, 5(1), 42–60. Preskill, H., & Torres, R. T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage. Rist, R. C., & Stame, N. (Eds.). (2006). From studies to streams: Managing evaluative systems. Piscataway, NJ: Transaction Publishers. Rodríguez-Campos, L. (2005). Collaborative evaluations: A step-by-step model for the evaluator. Tamarac, FL: Llumina Press. Rogers, P., Hacsi, T., Petrosino, A., & Huebner, T. (Eds.). (2000). Program theory in evaluation: Challenges and opportunities. New Directions for Evaluation, 87, 1–115. Rohs, F. R. (1999). Response shift bias: A problem in evaluating leadership development with self-report pretest-posttest measures. Journal of Agricultural Education, 40(4), 28–37. Rohs, F. R., & Langone, C. A. (1997). Increased accuracy in measuring leadership impacts. Journal of Leadership Studies, 4(1), 150–158. Ryan, K., & Schwandt, T. A. (Eds.). (2002). Exploring evaluator role and identity (Evaluation and Society Series). Greenwich, CT: Information Age Press. Sawyer, D. (1987). Analyzing pseudo–time series interventions. Evaluation Review, 11(2), 178–196. Schwandt, T. A. (1997). Evaluation as practical hermeneutics. Evaluation, 3(1), 69–83. Scriven, M. (1996). The theory behind practical evaluation. Evaluation, 2(4), 393–404. Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Thousand Oaks, CA: Sage. Shaw, I. F., Greene, J. C., & Mark, M. M. (2006). The SAGE handbook of evaluation. London, UK: SAGE Publications, Ltd.
122
bsource.indd 122
Evaluating the Impact of Leadership Development
2/1/08 4:35:16 PM
Smith, A. (1993). Management development evaluation and effectiveness. Journal of Management Development, 12(1), 20–32. Smith, M. E., & Brandenburg, D. C. (1991). Summative evaluation. Performance Improvement Quarterly, 4(2), 35–58. Sonnichsen, R. C. (2000). High impact internal evaluation: A practitioner’s guide to evaluating and consulting inside organizations. Thousand Oaks, CA: Sage. Stake, R. E. (1995). The art of case study research. Thousand Oaks, CA: Sage. Stecher, B. M., & Davis, W. A. (1987). How to focus an evaluation. Thousand Oaks, CA: Sage. Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey-Bass. Suarez-Balcazar, Y., & Harper, G. W. (2004). Empowerment and participatory evaluation in community intervention: Multiple benefits. Binghamton, NY: Haworth Press. Torres, R. T., Preskill, H. S., & Piontek, M. E. (1996). Evaluation strategies for communicating and reporting: Enhancing learning in organizations. Thousand Oaks, CA: Sage. Torres, R. T., Preskill, H. S., & Piontek, M. E. (1997). Communicating and reporting practices and concerns of internal and external evaluators. Evaluation Practice, 18(2), 105–125. Vicere, A. A. (1996). Executive education: The leading edge. Organizational Dynamics, 25(2), 67–81. Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (2004). Handbook of practical program evaluation (2nd ed.). San Francisco, CA: Jossey-Bass. Yin, R. (1984). Case study research: Design and methods. Thousand Oaks, CA: Sage.
Electronic Mail and Discussion Groups
EVALTALK. This group was established to provide a vehicle for open discussions concerning evaluation issues. Although it is sponsored by the American Evaluation Association (AEA), the list is available for anyone to use. To subscribe to EVALTALK, send an e-mail to [email protected]. The body of the message should read: SUBSCRIBE EVALTALK FirstName LastName. GOVTEVAL. This forum houses a discussion of government program evaluation. To subscribe to GOVTEVAL, send an e-mail to MAJORDOMO@NAIONET. NET. The body of the message should read: SUBSCRIBE GOVTEVAL [your e-mail address]. National Professional Organizations
Many of the organizations listed below have listservs and/or discussion forums that may be of interest. Contact the organizations directly to find out more about what they offer and how to join. Recommended Resources
bsource.indd 123
123
2/1/08 4:35:17 PM
Academy of Human Resource Development College of Technology Bowling Green State University Bowling Green, OH 43403 Phone: (419) 372-9155 http://www.ahrd.org/ Academy of Management P.O. Box 3020 Briarcliff Manor, NY 10510-8020 Phone: (914) 923-2607 http://aomonline.org/ American Educational Research Association 1230 17th Street, NW Washington, DC 20036-3078 Phone: (202) 223-9485 http://www.aera.net American Evaluation Association 16 Sconticut Neck Road, #290 Fairhaven, MA 02719 Phone: (888) 232-2275 http://www.eval.org American Management Association International 1601 Broadway New York, NY 10019 Phone: (212) 586-8100 http://www.amanet.org
124
bsource.indd 124
Evaluating the Impact of Leadership Development
2/1/08 4:35:17 PM
American Psychological Association 750 First Street, NE Washington, DC 20002-4242 Phone: (800) 374-2721, (202) 336-5500 http://www.apa.org American Psychological Society 1010 Vermont Avenue, NW Suite 1100 Washington, DC 20005-4918 Phone: (202) 783-2077 http://www.psychologicalscience.org American Society for Training and Development 1640 King Street, Box 1443 Alexandria, VA 22313-2043 Phone: (703) 683-8100 http://www.astd.org Australasian Evaluation Society Victoria University of Technology P.O. Box 5223 Lyneham ACT 2602 Australia Phone: 61 2 6262 9093 http://www.aes.asn.au
Recommended Resources
bsource.indd 125
125
2/1/08 4:35:17 PM
Canadian Evaluation Society La Société Canadienne d’Evaluation 1485 Laperriere Avenue Ottawa, ON K1Z 7S8 Phone: (613) 725-2526 http://evaluationcanada.ca/site.cgi?s=1&ss=4&_lang=EN International Society for Performance Improvement 1400 Spring Street, Suite 260 Silver Spring, MD 20910 Phone: (301) 587-8570 http://www.ispi.org National Council for Measurement in Education Central Office 1230 17th Street, NW Washington, DC 20036-3078 Phone: (202) 223-9318 http://www.ncme.org Psychometric Society 210 Curry Building P.O. Box 26171 University of NC at Greensboro Greensboro, NC 27402-6171 Phone: (336) 334-3474 http://www.psychometricsociety.org
126
bsource.indd 126
Evaluating the Impact of Leadership Development
2/1/08 4:35:18 PM
Society for Applied Sociology Department of Sociology, Anthropology, and Criminology Eastern Michigan University 712 Pray-Harrold, EMU Ypsilanti, MI 48197 Phone: (734) 487-0012 http://www.appliedsoc.org Society for Human Resource Management 1800 Duke Street Alexandria, VA 22314 Phone: (703) 548-3440 http://www.shrm.org Society for Industrial and Organizational Psychology SIOP Administrative Office 520 Ordway Avenue P.O. Box 87 Bowling Green, OH 43402-0087 Phone: (419) 353-0032 http://www.siop.org
Websites with Electronic Evaluation Resources
Educational Resources Information Center Clearinghouse on Assessment and Evaluation http://www.ericae.net/ Electronic Resources for Evaluators
www.luc.edu/faculty/eposava/resource.htm
Recommended Resources
bsource.indd 127
127
2/1/08 4:35:18 PM
The Evaluation Center at Western Michigan University
http://www.wmich.edu/evalctr/pubs/ecpub.htm
The Free Management Library
http://www.mapnp.org/library/
SocioNet Evaluation Links
http://www.socio.com/
W. K. Kellogg Foundation
http://www.wkkf.org/
128
bsource.indd 128
Evaluating the Impact of Leadership Development
2/1/08 4:35:18 PM
I N D E X
A
C
Action plans using evaluation findings to develop, 109–111 monitoring implementation of, 111 stakeholder role in developing, 110 See also Communicating results Adefuin, J., 5 Advice for Reporting Data (worksheet), 105e–107e Advocate role, 4 Archival documents/data, 93–96 Assessor evaluator role, 3
California Endowment, 5 Cao Yu, H., 5 Capacity builder role, 4 CCL (Center for Creative Leadership) how change is measured by, 46e on retrospective pretest and posttest assessments, 44 on 360-degree assessment retests, 45 Center for Problem-Oriented Policing, 42 Change based on evaluation findings determining needed, 107–108 designing evaluation to observe, 38–39 evaluation as tool for enhancing, 5 using evaluation finding to facilitate, 111–113 leadership development impact of social/systems, 16fig, 19 methods and issues of measuring, 43–48 Change measurement climate survey retests, 45–46 culture survey retests as, 46–47 degree-of-change ratings as, 44–45 how CCL approaches, 46e pre and post-initiative assessments as, 44
B Basic Elements of a Final Evaluation Report (worksheet), 104e–105e Behavioral observation evaluation using, 75–79 qualitative data example of, 78e quantitative data example of, 79e Bias definition of, 42e response-shift, 43, 44 Boundary spanner role, 4 Building Evaluation Capacity: 72 Activities for Teaching and Training (Preskill & Russ-Eft), 3
129
bindex.indd 129
2/1/08 4:30:57 PM
Change measurement (continued) response-shift bias problem of, 43, 44 retrospective pretest and posttest assessments as, 44 return on investment (ROI) as, 47–48 shifting perspectives challenges individual, 43e 360-degree assessment retests as, 45 Change survey, 71–74 Change Survey (worksheet examples), 72e–74e Checklists Communicating and Using Evaluation Findings, 114e Designing and Conducting the Evaluation, 99e Focusing the Evaluation, 35e Climate survey retests, 45–46 Communicating and Using Evaluation Findings: Checklist, 114e Communicating results Advice for Reporting Data worksheet, 105e–107e Basic Elements of a Final Evaluation Report worksheet, 104e–105e checklist on, 114e identifying specific courses of action, 107–109 mechanisms for, 102–103 what information and when to, 103 who receives information on, 103 See also Action plans Community data collection through statistics on, 90–93 determining leadership development expectations of, 16fig, 18 Concurrent evidence, 42e Confidentiality issue, 112 Construct validity, 42e
130
bindex.indd 130
Content validity, 42e Control groups, 39 Criterion validity, 42e Culture survey retests, 46–47
D Daily evaluation, 57–59 Data appropriate use of evaluation, 112e archival documents and, 93–96 confidentiality of individual, 112 qualitative, 32, 78e, 106e–107e quantitative, 32, 79e, 105e–106e storage and treatment of, 96–97 Data triangulation, 323 Data-collection methods archival documents and data, 93–96 behavioral observation, 75–79e change survey, 71–75 considering goal of evaluation for selecting, 33 considering purpose and use of evaluation for selecting, 32–33 daily evaluation, 57–59 end-of-initiative evaluation, 59–64 expectations survey, 55–57e focus group, 79–87 group dialogue, 87–90 guideline for using multiple, 38 interview, 64–69 learning survey, 69–71 matching evaluation questions to, 31–32 practical and feasibility issues of, 33–34 response rates to, 97e–98e workplace or community statistics, 90–93 See also Samples/sampling Day 1 Evaluation (worksheet), 58e–59e Degree-of-change ratings, 44–45
Index
2/1/08 4:30:58 PM
Designer evaluator role, 3–4 Designing and Conducting the Evaluation Checklist, 99e Developing Evaluation Questions (worksheet), 24e–25e Dialogue data-collection, 87–90 Documents data-collection using archival, 93–94, 96 example of archival, 94e–95e
E End-of-initiative evaluation, 59–64 End-of-Program (worksheet), 81e–84e Evaluation advantages of using both external/ internal, 2–3 common reasons for leadership development, 1 context for, 5 cyclical nature of the process, 4–5 daily, 57–59 end-of-initiative, 59–64 as enhancing change, 5 See also Evaluation findings; Leadership development Evaluation design guidelines for assessing change at different levels, 39 assessing multiple changes that can be observed, 38 use control groups for comparison, 39 for designing evaluation plan, 48–55 examining impact from multiple perspectives, 37–38 to look at change over time, 38–39 for measuring change, 43–48 use multiple-data-collection methods, 38 use pilot study, 40e for reliability and validity, 41e–42e
use time-series designs for comparison, 42 Evaluation findings Advice for Reporting Data, 105e–107e appropriate use of, 112e Basic Elements of a Final Evaluation Report, 104e–105e checklist on communicating, 114e used to create broader change and learning, 111–113 as data-collection method selection factor, 33 developing detailed action plan following, 109–110 effectively communicating, 102–107e four critical activities to complete, 101 identifying specific courses of action using, 107–109 monitoring action plan’s implementation, 111 planning how to use, 34 Evaluation focus checklist for, 35e defining purpose to establish, 10–15 determining impact over time, 19 determining resources for, 15 determining surface expectations, 19–20, 21e–22e determining types and amount of impact, 15–19 determining/prioritizing evaluation questions for, 23–31 identify and engage stakeholders for, 8–10e importance of establishing, 7–8 Evaluation plan design on data storage and treatment, 96–99e on data-collection methods, 55–96 elements of, 49
Index
bindex.indd 131
131
2/1/08 4:30:58 PM
Evaluation plan design (continued) example 1: ABC Program Longitudinal Outcome Evaluation, 50t–51t example 2: MNQ Program Pilot and Evaluation, 52t–54t importance of creating a good, 48–49 Evaluation questions creating focus through, 23 Developing Evaluation Questions (worksheet) for, 24e–25e examples of matching development initiatives to, 26 Focus Group (Several Months After the Program), 85e–86e Group Dialogue Questions (worksheet example), 89e guidelines for writing survey, 76e matching data-collection strategies to, 31–34 prioritizing, 31 target population for, 26–30e Evaluators external, 2–3 internal, 2–3 multiple roles played by, 3–4 EVALULead model, 20 Executive Sponsor Interview (worksheet), 68e Executive summary, 104 Expectations survey, 55–57e External evaluators, 2–3
F Fetterman, D. M., 15 Final Program Evaluation (worksheet), 60e–63e Findings. See Evaluation findings Focus Group (Several Methods After the Program) [worksheet], 85e–86e Focus groups data-collection using, 79–80, 86–87
132
bindex.indd 132
End-of-Program worksheet, 81e–84e Focus Group (Several Methods After the Program) worksheet, 85e–86e Focus. See Evaluation focus Focusing the Evaluation Checklist (worksheet), 35e
G Gerger, S. R., 44 Greenbaum, T. L., 80 Ground Rules for the Dialogue Method (worksheet), 88e Group dialogue, 87–90 Group Dialogue Questions (worksheet example), 89e Gulanick, N. A., 44
H The Handbook of Leadership Development Evaluation (Hannum, Martineau, & Reinelt), 5, 14, 20, 39, 47 Hannum, K. M., 1, 3, 5, 14, 18, 20, 39, 47 Howard, G. S., 44
I Inouye, T., 5 Internal consistency, 41e Internal evaluators, 2–3 Interrater agreement, 41e Interviews Executive Sponsor Interview worksheet, 68e guidelines on data-collection using, 64–65, 67, 69 Participant Interview worksheet, 66e–67e
J Joint Committee on Standards for Educational Evaluation, 37, 113
Index
2/1/08 4:30:59 PM
K Katzev, A., 44 Kraemer, H. C., 40 Krueger, R. A., 80
L Langone, C. A., 44 Leadership development determining types and amount of impact, 15–19 expectations for, 16fig, 19–20, 21e–22e list of possible outcomes for, 10 See also Evaluation Leadership development initiatives addressing surface expectations in, 19–20 evaluation findings used to broaden, 108, 111–113 exploring need for follow-up activity to, 109 matching evaluation questions to, 26 pre- and post-initiative assessments of, 44 retrospective pretest and posttest assessments of, 44 Learning using evaluation finding to facilitate, 111–113 surface expectations on, 19–20 Learning Opportunities (worksheet), 11e Learning survey, 69–71 Learning Survey (worksheet), 70e
M Managers of and Those Working with the Target Population (worksheet), 30e Martineau, J. W., 1, 3, 5, 14, 18, 20, 39, 47 Maxwell, S. E., 44
McGuigan, W., 44 McGuire, J. B., 87 Measuring change. See Change measurement Morgan, D. L., 80
N Nance, D. W., 44 Network building, 18
O Organizations climate survey retests of, 45–46 culture survey retests of, 46–47 determining leadership development expectations of, 16fig, 19–20, 21e–22e determining types and amount of impact on, 15–19
P Palus, C. J., 87 Participant Interview (worksheet), 66e–67e Participants confidentiality of individual, 112 interviewing, 66e–67e response rates of, 97e–98e samples of, 31–32 Patton, M. Q., 14 Peterson, D. B., 45 Phillips, J. J., 90 Pilot Study Is a Good Evaluation Tool (worksheet), 40e Plan. See Evaluation plan design Planner evaluator role, 3–4 Post-initiative assessment, 44 Pratt, C., 44 Pre-initiative assessment, 44 Preprogram Expectations Survey Example (worksheet), 56e–57e
Index
bindex.indd 133
133
2/1/08 4:30:59 PM
Preskill, H., 3, 5, 101 Purpose evaluation focus by establishing, 10–12 Purpose Definition (worksheet), 12e–14e
Q Qualitative data advice for reporting, 106e–107e behavioral observation, 78e data-collection method using, 32 Quantitative data advice for reporting, 105e–106e behavioral observation, 79e data-collection method using, 32 Questions. See Evaluation questions
deciding who receives evaluation findings, 103 determining leadership development impact on, 15–19 gathering and consulting on evaluation findings, 109 identifying and engaging, 8–10e Surface Expectations (worksheet), 21e–22e Surveys change, 71–75 expectations, 55–57e guidelines for writing questions, 76e learning, 69–71 workplace, 91e–92e
T R
134
bindex.indd 134
Ralph, K. M., 44 Reflective practitioner, 4 Reinelt, C., 1, 3, 5, 14, 18, 20, 39, 48 Reliability and Validity (worksheet), 41e–42e Resource determination, 15 Response rates, 97e–98e Response-shift bias, 43, 44 Retrospective pretest assessment, 44 Return on investment (ROI), 47–48 Rohs, F. R., 44 Russ-Eft, D., 3
Target population importance of understanding your, 26 managers of and those working with, 29, 30e Target Population (worksheet) on, 27e–28e Teams/groups, 16fig, 17 Thiemann, S., 40 360-degree assessment retests, 45 Time-series designs, 42 Torres, R. T., 5, 101 Trainer evaluator role, 4 Translator role, 4
S
V
Samples/sampling, 31–32 See also Data-collection methods Sawyer, D., 42 Social/systems change, 16fig, 19 Stakeholder Identification (worksheet), 9e–10e Stakeholders action plan development role of, 110
Validity, 41e–42e
W Wandersman, A., 15 Websites California Endowment, 5 Center for Problem-Oriented Policing, 42
Index
2/1/08 4:31:00 PM
on leadership development initiatives, 20 list of possible outcomes for leadership development, 10 time-series designs, 42 W.K. Kellogg Foundation, 14 W.K. Kellogg Foundation, 14 Workplace statistics, 90–93 Workplace Statistics Survey (worksheet example), 91e–92e Worksheets Advice for Reporting Data, 105e–107e Basic Elements of a Final Evaluation Report, 104e–105e Behavioral Observation: Qualitative Data, 78e Behavioral Observation Rating Table: Quantitative Data, 79e Change Survey examples, 72e–74e Communicating and Using Evaluation Findings: Checklist, 114e Day 1 Evaluation, 58e–59e Developing Evaluation Questions, 24e–25e End-of-Program, 81e–84e
Executive Sponsor Interview, 68e Final Program Evaluation, 60e–63e Focus Group (Several Methods After the Program), 85e–86e Focusing the Evaluation Checklist, 35e Ground Rules for the Dialogue Method, 88e Group Dialogue Questions (example), 89e Learning Opportunities, 11e Learning Survey, 70e Managers of and Those Working with the Target Population, 30e Participant Interview, 66e–67e Pilot Study Is a Good Evaluation Tool, 40e Preprogram Expectations Survey Example, 56e–57e Purpose Definition, 12e–14e Reliability and Validity, 41e–42e Stakeholder Identification, 9e–10e Surface Expectations, 21e–22e Target Population, 27e–28e Workplace Statistics Survey (example), 91e–92e
Index
bindex.indd 135
135
2/1/08 4:31:01 PM
bindex.indd 136
2/1/08 4:31:02 PM
A BO U T
T H E
A U T H O R S
K
elly M. Hannum is an enterprise associate in the Global Leadership and Diversity and Design and Evaluation groups at the Center for Creative Leadership. Her work focuses on evaluation and leadership development across different cultures. She coauthored The Handbook of Leadership Development Evaluation and “Evaluation in North America” in Cross-Cultural Evaluation: An International Annotated Bibliography. She holds a Ph.D. in educational research, measurement, and evaluation from the University of North Carolina at Greensboro.
J
ennifer W. Martineau serves as director of the Center for Creative Leadership’s Design and Evaluation Center. At CCL since 1993 and with more than twenty years’ experience in the field of evaluation, she has focused her attention on the evaluation of leadership development programs and initiatives. She coauthored The Handbook of Leadership Development Evaluation, and her work can also be found in book chapters, peer-reviewed journals, and practitioneroriented publications. She holds a Ph.D. in industrial and organizational psychology from the Pennsylvania State University. 137
babout.indd 137
2/1/08 4:30:22 PM
Pfeiffer Publications Guide This guide is designed to familiarize you with the various types of Pfeiffer publications. The formats section describes the various types of products that we publish; the methodologies section describes the many different ways that content might be provided within a product. We also provide a list of the topic areas in which we publish.
FORMATS In addition to its extensive book-publishing program, Pfeiffer offers content in an array of formats, from fieldbooks for the practitioner to complete, ready-to-use training packages that support group learning.
FIELDBOOK Designed to provide information and guidance to practitioners in the midst of action. Most fieldbooks are companions to another, sometimes earlier, work, from which its ideas are derived; the fieldbook makes practical what was theoretical in the original text. Fieldbooks can certainly be read from cover to cover. More likely, though, you’ll find yourself bouncing around following a particular theme, or dipping in as the mood, and the situation, dictate.
HANDBOOK A contributed volume of work on a single topic, comprising an eclectic mix of ideas, case studies, and best practices sourced by practitioners and experts in the field. An editor or team of editors usually is appointed to seek out contributors and to evaluate content for relevance to the topic. Think of a handbook not as a ready-to-eat meal, but as a cookbook of ingredients that enables you to create the most fitting experience for the occasion. RESOURCE Materials designed to support group learning. They come in many forms: a complete, ready-to-use exercise (such as a game); a comprehensive resource on one topic (such as conflict management) containing a variety of methods and approaches; or a collection of like-minded activities (such as icebreakers) on multiple subjects and situations. TRAINING PACKAGE An entire, ready-to-use learning program that focuses on a particular topic or skill. All packages comprise a guide for the facilitator/trainer and a workbook for the participants. Some packages are supported with additional media— such as video—or learning aids, instruments, or other devices to help participants understand concepts or practice and develop skills.
both01.indd 138
2/1/08 4:33:58 PM
• Facilitator/trainer’s guide Contains an introduction to the program, advice on how to organize and facilitate the learning event, and step-by-step instructor notes. The guide also contains copies of presentation materials—handouts, presentations, and overhead designs, for example—used in the program. • Participant’s workbook Contains exercises and reading materials that support the learning goal and serves as a valuable reference and support guide for participants in the weeks and months that follow the learning event. Typically, each participant will require his or her own workbook.
ELECTRONIC CD-ROMs and web-based products transform static Pfeiffer content into dynamic, interactive experiences. Designed to take advantage of the searchability, automation, and ease-of-use that technology provides, our e-products bring convenience and immediate accessibility to your workspace.
METHODOLOGIES CASE STUDY A presentation, in narrative form, of an actual event that has occurred inside an organization. Case studies are not prescriptive, nor are they used to prove a point; they are designed to develop critical analysis and decision-making skills. A case study has a specific time frame, specifies a sequence of events, is narrative in structure, and contains a plot structure—an issue (what should be/have been done?). Use case studies when the goal is to enable participants to apply previously learned theories to the circumstances in the case, decide what is pertinent, identify the real issues, decide what should have been done, and develop a plan of action. ENERGIZER A short activity that develops readiness for the next session or learning event. Energizers are most commonly used after a break or lunch to stimulate or refocus the group. Many involve some form of physical activity, so they are a useful way to counter postlunch lethargy. Other uses include transitioning from one topic to another, where “mental” distancing is important.
EXPERIENTIAL LEARNING ACTIVITY (ELA) A facilitator-led intervention that moves participants through the learning cycle from experience to application (also known as a Structured Experience). ELAs are carefully thought-out designs in which there is a definite learning purpose and intended outcome. Each step—everything that participants do during the activity—facilitates the accomplishment of the stated goal. Each ELA includes complete instructions for facilitating the intervention and a clear statement of goals, suggested group size and timing, materials required, an explanation of the process, and, where appropriate, possible variations to the activity. (For more detail on Experiential
both01.indd 139
2/1/08 4:33:59 PM
Learning Activities, see the Introduction to the Reference Guide to Handbooks and Annuals, 1999 edition, Pfeiffer, San Francisco.)
GAME A group activity that has the purpose of fostering team spirit and togetherness in addition to the achievement of a pre-stated goal. Usually contrived—undertaking a desert expedition, for example—this type of learning method offers an engaging means for participants to demonstrate and practice business and interpersonal skills. Games are effective for team building and personal development mainly because the goal is subordinate to the process—the means through which participants reach decisions, collaborate, communicate, and generate trust and understanding. Games often engage teams in “friendly” competition.
ICEBREAKER A (usually) short activity designed to help participants overcome initial anxiety in a training session and/or to acquaint the participants with one another. An icebreaker can be a fun activity or can be tied to specific topics or training goals. While a useful tool in itself, the icebreaker comes into its own in situations where tension or resistance exists within a group.
INSTRUMENT A device used to assess, appraise, evaluate, describe, classify, and summarize various aspects of human behavior. The term used to describe an instrument depends primarily on its format and purpose. These terms include survey, questionnaire, inventory, diagnostic, survey, and poll. Some uses of instruments include providing instrumental feedback to group members, studying here-and-now processes or functioning within a group, manipulating group composition, and evaluating outcomes of training and other interventions. Instruments are popular in the training and HR field because, in general, more growth can occur if an individual is provided with a method for focusing specifically on his or her own behavior. Instruments also are used to obtain information that will serve as a basis for change and to assist in workforce planning efforts. Paper-and-pencil tests still dominate the instrument landscape with a typical package comprising a facilitator’s guide, which offers advice on administering the instrument and interpreting the collected data, and an initial set of instruments. Additional instruments are available separately. Pfeiffer, though, is investing heavily in e-instruments. Electronic instrumentation provides effortless distribution and, for larger groups particularly, offers advantages over paper-and-pencil tests in the time it takes to analyze data and provide feedback.
LECTURETTE A short talk that provides an explanation of a principle, model, or process that is pertinent to the participants’ current learning needs. A lecturette is intended to establish a common language bond between the trainer and the participants by providing a mutual frame of reference. Use a lecturette as an introduction to a group activity or event, as an interjection during an event, or as a handout.
both01.indd 140
2/1/08 4:33:59 PM
MODEL A graphic depiction of a system or process and the relationship among its elements. Models provide a frame of reference and something more tangible, and more easily remembered, than a verbal explanation. They also give participants something to “go on,” enabling them to track their own progress as they experience the dynamics, processes, and relationships being depicted in the model.
ROLE PLAY A technique in which people assume a role in a situation/scenario: a customer service rep in an angry-customer exchange, for example. The way in which the role is approached is then discussed and feedback is offered. The role play is often repeated using a different approach and/or incorporating changes made based on feedback received. In other words, role playing is a spontaneous interaction involving realistic behavior under artificial (and safe) conditions.
SIMULATION A methodology for understanding the interrelationships among components of a system or process. Simulations differ from games in that they test or use a model that depicts or mirrors some aspect of reality in form, if not necessarily in content. Learning occurs by studying the effects of change on one or more factors of the model. Simulations are commonly used to test hypotheses about what happens in a system— often referred to as “what if?” analysis—or to examine best-case/worst-case scenarios.
THEORY A presentation of an idea from a conjectural perspective. Theories are useful because they encourage us to examine behavior and phenomena through a different lens.
TOPICS The twin goals of providing effective and practical solutions for workforce training and organization development and meeting the educational needs of training and human resource professionals shape Pfeiffer’s publishing program. Core topics include the following: Leadership & Management Communication & Presentation Coaching & Mentoring Training & Development E-Learning Teams & Collaboration OD & Strategic Planning Human Resources Consulting
both01.indd 141
2/1/08 4:34:00 PM
The Handbook of Leadership Development Evaluation Kelly M. Hannum • Jennifer W. Martineau Claire Reinelt, Eds. ISBN 978-07879-8217-1 • Cloth Available wherever books are sold www.josseybass.com "This inspiring book fills a critical void in the leadership development field. Its diversity of evaluation content, authors, and approaches makes it an invaluable resource for anyone involved in designing, implementing, and evaluating leadership development programs. A must-have!"—Hallie Preskill, professor, School of Behavioral and Organizational Sciences, Claremont Graduate University, and president-elect, American Evaluation Association "With so many funders using leadership development as a strategy, this book couldn't have come at a better time. More than a collection of individual chapters, this volume presents synthetic collaboration among the authors and the editors to explore how a variety of evaluation strategies are critical to the design, implementation, and outcomes of leadership programs. It is an important contribution to the field."—Constance Pechura, executive director, Treatment Research Institute Sponsored by the Center for Creative Leadership, this Handbook of Leadership Development Evaluation brings together a distinguished list of contributors who share their knowledge and expertise about this important field. With the increase in the number of organizational leadership development programs, there is a pressing need for evaluation to answer important questions, improve practice, and inform decisions. The Handbook is a comprehensive resource filled with examples, tools, and the most innovative models and approaches designed to evaluate leadership development in a variety of settings. It will help you answer the most common questions about leadership development efforts, including:
x x x x x
What difference does leadership development make? What development and support strategies work best to enhance leadership? Is the time and money spent on leadership development worthwhile? What outcomes can be expected from leadership development? How can leadership development efforts be sustained?
This groundbreaking handbook will help practitioners and researchers lay a foundation for the open exchange of ideas and put in place the process of application, adaptation, and subsequent learning of leadership development evaluation.
badvert.indd 142
2/1/08 4:50:26 PM
