June)

from the editor Editor in Chief: Steve McConnell

■

Construx Software

■

[email protected]

An Ounce of Prevention Steve McConnell

A

“

stitch in time saves nine,” the old saying goes. “An ounce of prevention is worth a pound of cure.” In software, these expressions translate into the common observation that the longer a defect stays in process, the more expensive it is to fix.1 Industry reports about the magnitude of the cost increase have varied over the years. The highest ratio I’ve seen published came from Barry Boehm and Philip Papaccio in 1988.2 They reported that requirements defects that made their way into the field could cost 50 to 200 times as much to correct as defects that were corrected close to the point of creation. Of course, “50 to 200 times” is a rough average, and in the worst cases, the sky is the limit for defect costs—literally. The US space program had two highprofile failures in 1999: in both, correcting a defect “in the field” was not possible, and the software errors that went undetected until the software was in the field ended up costing hundreds of millions of dollars. I’ve previously presented a rough rule of thumb that early, upstream defects generally cost 10 to 100 times as much to remove late downstream as they do to remove close to the point where they are created.1 These observations have been used to justify a focus on upstream quality assurance activities such as extensive requirements work, design work, and technical reviews. These old sayings and rules of thumb have come under attack in recent years. Some people claim that software defects aren’t as expensive to correct as they used to be; costs don’t increase as quickly as they used to. In other words, an ounce of prevention is not worth a pound of cure, but Copyright © 2001 Steven C. McConnell. All Rights Reserved.

perhaps only an ounce of cure.3 Some claim that we are expending more effort on prevention than we would by fixing the defects later—that we’re spending a pound of prevention to avoid an ounce of cure.

Old support for an old saying One common project dynamic is to cut corners because “we’re only 30 days from shipping.” If you’re in a hurry, for example, you might decide that you don’t have time to design and code a separate, completely clean printing module. So you piggyback printing onto the screen display module. You know that’s a bad design that won’t be extensible or maintainable, but you don’t have time to do the right design. Three months later, when the product still hasn’t shipped, those cut corners come back to haunt you. You find that the people using the prerelease software are unhappy with printing, and the only way to satisfy their requests is to significantly extend the printing functionality, which can’t be done with the piggybacked version. Unfortunately, in the three months since you took the shortcut, the printing functionality and the screen display functionality have become thoroughly intertwined. Redesigning printing and separating it from the screen display is now a tough, time-consuming, errorprone operation. We have understood the dynamic in play in this example at least since the 1970s when IBM observed that software quality and software schedules were related. It found that the products with the lowest defect counts were also the products with the shortest schedules.4 Work on a software project generally follows a pattern of a small number of high-leverage upstream decisions providing the basis for a much larger number of lower-leverage downstream May/June 2001

IEEE SOFTWARE

5

FROM THE EDITOR

D E PA R T M E N T E D I T O R S

Bookshelf: Warren Keuffel, [email protected] Country Report: Deependra Moitra, [email protected] Design: Martin Fowler, ThoughtWorks, [email protected] Loyal Opposition: Robert Glass, Computing Trends, [email protected] Manager: Don Reifer, Reifer Consultants, [email protected] Quality Time: Jeffrey Voas, Cigital, [email protected] STAFF

Group Managing Editor Crystal Chweh Senior Lead Editor Dale C. Strok [email protected] Associate Lead Editors Jenny Ferrero and Dennis Taylor Staff Lead Editor Shani Murray Staff Editor Scott Lorenz Andresen Magazine Assistant Dawn Craig [email protected] Art Director Toni Van Buskirk Cover Illustration Dirk Hagner Technical Illustrator Alex Torres Production Artists Carmen Flores-Garvey and Larry Bauer Acting Executive Director Anne Marie Kelly Publisher Angela Burgess Membership/Circulation Marketing Manager Georgann Carter Advertising Assistant Debbie Sims CONTRIBUTING EDITORS

Greg Goth, Kirk Kroeker, Nancy Mead, Ware Myers, Judy Shane, Margaret Weatherford

Editorial: All submissions are subject to editing for clarity, style, and space. Unless otherwise stated, bylined articles and departments, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IEEE Software does not necessarily constitute endorsement by the IEEE or the IEEE Computer Society. To Submit: Send 2 electronic versions (1 word-processed and 1 postscript or PDF) of articles to Magazine Assistant, IEEE Software, 10662 Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720-1314; [email protected]. Articles must be original and not exceed 5,400 words including figures and tables, which count for 200 words each.

6

IEEE SOFTWARE

May/ June 2001

decisions. Thus we make high-leverage requirements decisions that provide the basis for medium-leverage design decisions, which in turn provide the basis for low-leverage code, test-case, and end-user-documentation decisions. A small mistake in upstream work can affect large amounts of downstream work. A change to a single sentence in a requirements specification can imply changes in hundreds of lines of code spread across numerous classes or modules, dozens of test cases, and numerous pages of end-user documentation. Capers Jones reports that reworking defective requirements, design, and code typically consumes 40 to 50 percent or more of the total cost of most software projects and is the single largest cost driver.5 Tom Gilb reports that about half of all defects usually exist at design time,6 which is confirmed by Jones’s data. If half of all defects are upstream defects, you should be able to save effort by detecting defects earlier than system testing. Jones reports that, as a rule of thumb, every hour you spend on technical reviews upstream will reduce your total defect repair time from three to ten hours; that is, one ounce of prevention is worth three to ten ounces of cure.7 Has this dynamic changed in recent years? Recent data from Hughes Aircraft shows that the average requirements defect still takes 10 times as much effort to correct during system testing as it does during requirements analysis.8 The dynamics of defect-cost increase are inherent in the nature of software engineering work. It doesn’t matter whether the project follows an oldfashioned waterfall life-cycle model or uses a cutting-edge iterative approach— design, code, test cases, and documentation will have dependencies upon requirements regardless of whether the project is done all at once or divided into numerous incremental releases. Overall, I see no indication either from industry data or analysis that the dynamics of defect-cost increase have changed in recent years.

What does that ounce of prevention look like? While the underlying dynamic of defect-cost increase has not changed,

our understanding of how to detect upstream defects has improved considerably. Not too many years ago, we thought that the best way to detect requirements defects was to capture an exhaustive set of requirements in a monolithic requirements specification and then to subject that specification to intensive reviews. Although industry data suggests that this approach is cost-effective compared to the alternative of jumping straight into coding and then fixing requirements defects at construction time, we now know of numerous alternatives that are often preferable to the monolithic-requirementsspecification approach: ■ Involve end users as early as possible.

Several studies have found that enduser involvement is key to stable requirements and software project success.9 ■ Create a throwaway prototype. Create a throwaway UI and put the prototype in front of real end users. Get feedback. Revise the prototype until the user is excited about the system. Then build the system. This approach correlates with good requirements stability and low system cost.5 ■ Deliver the software incrementally. Write production code for a small amount of the system. Put that functionality in front of the user. Revise the requirements, design, and code until the user is excited about the system. This approach does not entirely eliminate the defect-cost increase dynamic, but it shortens the feedback loop from requirements to user feedback in a way that reduces the number of downstream dependences that will be based on erroneous upstream work. This sort of incremental delivery approach correlates with high user satisfaction and lower total development costs.1,6 ■ Conduct a requirements workshop. Fast requirements elicitation techniques such as joint application development sessions are an effective way to shorten the time required to collect accurate requirements while simultaneously reducing requirements volatility downstream.5

FROM THE EDITOR ■ Perform use case analysis. Rather

than being satisfied with the users’ first explanation of what they want a system to do, examine the system’s expected usage patterns to better understand users’ real needs. ■ Create the user manual first. Some organizations have had good success creating their user manuals as a substitute for or supplement to a traditional requirements specification. End users seem to be better able to understand the contents of a user manual than a traditional requirements specification, and requirements elicitation goes more smoothly.

many good techniques have emerged in the past few years.

EDITOR IN CHIEF: Steve McConnell 10662 Los Vaqueros Circle Los Alamitos, CA 90720-1314 [email protected] EDITOR IN CHIEF EMERITUS: Alan M. Davis, Omni-Vista

References

New twists on old sayings Software engineering advances by periodically reexamining questions that we think we’ve already answered. An ounce of prevention is still generally worth a pound of cure, but some recent developments have improved the “ounces of prevention” at our disposal. I find it encouraging that so

Upcoming

1. S. McConnell, Rapid Development, Microsoft Press, Redmond, Wash., 1996. 2. B.W. Boehm and P.N. Papaccio, “Understanding and Controlling Software Costs,” IEEE Trans. Software Eng., vol. 14, no. 10, Oct. 1988, pp. 1462–1477. 3. K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, Reading, Mass., 2000. 4. C. Jones, Applied Software Measurement: Assuring Productivity and Quality, 2nd ed., McGraw-Hill, New York, 1997. 5. C. Jones, Estimating Software Costs, McGraw-Hill, New York, 1998. 6. T. Gilb, Principles of Software Engineering Management, Addison-Wesley, Wokingham, U.K., 1988. 7. C. Jones, Assessment and Control of Software Risks, Yourdon Press, Englewood Cliffs, N.J., 1994. 8. R.R. Willis et al., Hughes Aircraft’s Widespread Deployment of a Continuously Improving Software Process, tech. report CMU/SEI-98-TR-006, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 1998. 9. Charting the Seas of Information Technology, tech. report, The Standish Group, Dennis, Mass., 1994.

Topics

July/August ‘01: Fault Tolerance September/October ‘01: Benchmarking Software Organizations November/December ‘01: Extreme Programming Update

A S S O C I AT E E D I T O R S I N C H I E F

Design: Maarten Boasson, Quaerendo Invenietis [email protected] Construction: Terry Bollinger, Mitre Corp. [email protected] Requirements: Christof Ebert, Alcatel Telecom [email protected] Management: Ann Miller, University of Missouri, Rolla [email protected] Quality: Jeffrey M. Voas, Cigital [email protected] EDITORIAL BOARD

Don Bagert, Texas Tech University Andy Bytheway, Univ. of the Western Cape Richard Fairley, Oregon Graduate Institute Martin Fowler, ThoughtWorks Robert Glass, Computing Trends Natalia Juristo, Universidad Politécnica de Madrid Warren Keuffel, independent consultant Brian Lawrence, Coyote Valley Software Karen Mackey, Cisco Systems Stephen Mellor, Project Technology Deependra Moitra, Lucent Technologies, India Don Reifer, Reifer Consultants Wolfgang Strigel, Software Productivity Centre Karl Wiegers, Process Impact INDUSTRY ADVISORY BOARD

Robert Cochran, Catalyst Software, chair Annie Kuntzmann-Combelles, Q-Labs Enrique Draier, PSINet Eric Horvitz, Microsoft Research David Hsiao, Cisco Systems Takaya Ishida, Mitsubishi Electric Corp. Dehua Ju, ASTI Shanghai Donna Kasperson, Science Applications International Pavle Knaflic, Hermes SoftLab Günter Koch, Austrian Research Centers Wojtek Kozaczynski, Rational Software Corp. Tomoo Matsubara, Matsubara Consulting Masao Matsumoto, Univ. of Tsukuba Dorothy McKinney, Lockheed Martin Space Systems Susan Mickel, AgileTV Dave Moore, Vulcan Northwest Melissa Murphy, Sandia National Laboratories Kiyoh Nakamura, Fujitsu Suzanne Robertson, Altantic Systems Guild Grant Rule, Software Measurement Services Girish Seshagiri, Advanced Information Services Chandra Shekaran, Microsoft Martyn Thomas, Praxis Rob Thomsett, The Thomsett Company John Vu, The Boeing Company Simon Wright, Integrated Chipware Tsuneo Yamaura, Hitachi Software Engineering M A G A Z I N E O P E R AT I O N S C O M M I T T E E

January/February ‘02: Building Security from the Ground Up March/April ‘02: The Engineering of Internet Software

Sorel Reisman (chair), James H. Aylor, Jean Bacon, Thomas J. Bergin, Wushow Chou, William I. Grosky, Steve McConnell, Daniel E. O’Leary, Ken Sakamura, Munindar P. Singh, Francis Sullivan, James J. Thomas, Yervant Zorian P U B L I C AT I O N S B O A R D

Rangachar Kasturi (chair), Angela Burgess (publisher), Jake Aggarwal, Laxmi Bhuyan, Mark Christensen, Lori Clarke, Mike T. Liu, Sorel Reisman, Gabriella Sannitti di Baja, Sallie Sheppard, Mike Williams, Zhiwei Xu

May/June 2001

IEEE SOFTWARE

7

manager Editor: Donald J. Reifer

■

Reifer Consultants

■

[email protected]

Software Engineering and the Law John Cosgrove, P.E.

F

or good or otherwise, the legal system has discovered the world of computers and its practitioners. Anyone opening a daily newspaper knows that litigation involving computers and software has exploded in recent years. On balance, the net effect of this attention might be positive because it gives practitioners an economic incentive to improve the way we work. Indeed, lawyers might well be the ones who provide the incentives for realistic contractual commitments, worst-case software engineering development practices, and a total organizational commitment to quality. Something similar happened to the US automobile industry, benefiting both carmakers and the public. The threat Tom DeMarco and Tim Lister estimate that “costs of litigation are rising faster than any other aspect of software development,” and “[l]itigation costs are … a larger component than coding.”1 The legal system has not overlooked computing’s pervasive presence in every aspect of society. In the recent flood of computer-related litigation, software forensic consulting in particular has multiplied in recent years, as DeMarco and Lister also noted. Usually, this involves disputes over computer projects and contracts but often requires an expert opinion in unlikely matters. Recent forensic clients have included a divorce dispute needing an economic evaluation of a software product, a 14

IEEE SOFTWARE

May/June 2001

wrongful termination involving computer system crashes, production of evidence of gambling during business hours, and academic plagiarism charges. Resolving all these disputes required computer expertise. More important are the computer-related issues that threaten our economic structure and our citizens’ health and safety. These threats have been with us for some time, but only recently has the legal system identified the litigation potential of computers and software. Why software is different Most engineered systems start with comprehensive plans and specifications. Few software-intensive systems do! This simple fact sets the stage for most of the issues leading to litigation. In fact, it is usually impossible to completely define most practical software systems. Watts Humphrey stated the dilemma: “With software the challenge is to balance the unknowable nature of the requirements with the business need for a firm contractual relationship.”2 Thus, there is no clear answer to the inevitable legal ambiguities. Both parties must learn to live with these ambiguities and cooperatively resolve each issue in a timely manner. When this understanding breaks down, litigation results, and the ultimate resolution is costly for both parties. DeMarco and Lister titled their article “Both Sides Always Lose: Litigation of Software-Intensive Contracts.” The challenge as a software professional is to steer the parties away from this disastrous state. 0740-7459/01/$10.00 © 2001 IEEE

MANAGER

Explaining the unexplainable

ects. Litigation has stimulated many manufacturers to apply worst-case design principles to their engineering practices. Thus, car manufacturers can defend themselves by citing the extensive research and testing they’ve undertaken to validate their designs before product release. Although not yet perfect, major improvements in safety and reliability have resulted. Software engineering likely will soon feel the same pressure.

As the wit said of computerintensive technical claims: “All the parties are lying, but none of them knows it.” That’s doubly true of legal discourse involving computers. People have become so accustomed to asserting the most unsupportable conclusions from computer “facts” that they come to believe that almost anything can be true sometimes, so they might as well claim it. Because the complexity is usually very high, it is exceedingly difficult to “prove” any assertion false in a typical legal proceeding. What to do There are no guarantees, but if the record of the system development process shows “all reasonable steps,” this is the best defense possible. Even though a well-documented process is no guarantee of quality, high quality and consistent results are almost always a result of a well-conceived, and usually well-documented, process. At least, the accusation of negligence is unlikely to hold if this is done. Also, the performance of the steps should be recorded. I’ll describe the method of defining the reasonable steps in a moment. Avoiding and surviving litigation Several years ago, Bud Lawson proposed a method to define the engineering processes used to develop software-intensive, safety-critical systems. Simply stated, the method “assumes—a priori—that legal action has been brought against them for the product that they are about to produce.”3 Then, “all reasonable steps” must be present in the engineering activities to defend against the action. DeMarco and Lister suggest a similar strategy in that “[t]he things you do to win a litigation … also are … the principal things that you should do to avoid litigation.” Simply stated, good engineering in the best sense is the best legal defense. Typically, a software development

team’s culture is seldom driven by practices that a lawyer could defend as “all reasonable steps” in court. Real-life projects are defined by needs that are often independent of any achievable means. Humphrey’s Why Software Organizations Are Chaotic describes a project: “the schedule … represented what was needed and had nothing to do with an achievable plan to make it work.”2 Even if a software-development team meets the schedule, it has made so many compromises that the quality is usually unacceptable, establishing a different basis for litigation. The solution is to insist on achievable expectations that enable the team to engineer the system according to the “all reasonable steps” principle. As one approach, software developers might apply worst-case design principles to their development proj-

As the wit said of computer-intensive technical claims: “All the parties are lying, but none of them knows it.”

Managing expectations Humphrey also expressed a cultural weakness concerning unrealistic expectations: “[D]irected by top management to run a mile in two minutes, what should they do? … many programmers start looking for their running shoes.”2 As long as this response continues, “reasonable” cannot often be truthfully applied to software development. A recent case involved a customer who expected delivery of a “state-ofthe-art” financial system in six months or less. The customer knew that the undertaking was unprecedented (never before accomplished), large (over 500K lines of code), critical (mistakes risking millions of dollars a day), and ill-defined (tens of communication interfaces changing constantly). Even so, the buyer terminated the contract and sued the supplier when it could not meet those expectations. The expert’s role was to explain the reality of what the opposing side expected and the implications of the constant changes that the customer imposed on the developer during the project’s life. Soon afterwards, the case settled. These types of expectations are not as rare as they should be. Many argue that some lack of realism is the norm in software development. It is safe to say that, when unrealistic expectations are left alone, litigation will likely follow. Software developers avoid the issue because the May/June 2001

IEEE SOFTWARE

15

MANAGER

IEEE

FOR

CALL Articles Software Engineering of Internet Software

In less than a decade, the Internet has grow n from a little-known back road of nerds into a central highway for worldwide comm erce, information, and entertainment. This shift has introduced a new language. We speak of Internet time, Internet software, and the rise and fall of e-business. Essential to all of this is the software that makes the Internet work. From the infrastructure companies that create the tools on which e-business runs to the Web design boutiques that deploy slick Web sites using the latest technology, software lies behin d the shop windows, newspapers, and bank notes . How is this new Internet software differ ent than the software created before every thing became e-connected? Are the tools different? Are the designs different? Are the processes different? And have we forgo tten important principles of software engin eering in the rush to stake claims in the new Webi fied world? We seek original articles on what it mean s to do Internet software. Specific quest ions that might be addressed include (but are not limited to): • How do we apply engineering techniques to Internet software? • Is it all “code and fix” or can we apply a broader variety of effective practices? • How have Internet time and new techn ologies affected the culture of software development organizations? • What role does technology play? • How does user interface design chang e for a Web interface? How does it stay the same? • What are the design principles and patte rns behind effective Internet software? • What are the real problems and soluti ons behind data interchange on the globa l Internet scale? • How are Internet software developme nt and testing practices different from other kinds of software development? Authors should present their work in terms that are useful to the software community at large, emphasizing lessons learned from practical experience. Please submit two electronic copies, one in RTF or Microsoft Word and one in PDF or Postscript, by 15 August 2001. Articl es should be 4–12 double-spaced pages, including illustrations, or 2,800–5,400 word s, with each illustration, graph, or table count ing as 200 words. Submissions are peer-reviewed and are subject to editin g for style, clarity, and space. For detai led author guidelines, see computer.o rg/ software/author.htm or contact the magazine assistant Dawn Craig at [email protected]. Guest Editors:

Publication: March/April 2002 Submission deadline: 15 August 2001 16

IEEE SOFTWARE

May/June 2001

Elisabeth Hendrickson, Quality Tree Software, Inc. 7563 Cottonwood Lane, Pleasanton, CA 94588, USA [email protected] Martin Fowler, Chief Scientist, Thoughtworks 651 W Washington Blvd, Suite 500, Chicago, IL 60661, USA [email protected]

process of educating the customer is always painful and often fatal to keeping the contract going. The alternative—litigation—might be worse. The only solution is the painful process of confronting each perception in a patient, orderly way and documenting the mutual understanding or unresolved questions. It is only “good engineering” to insist on defining a project in achievable terms. Warning Litigation—potential or otherwise—involving computers and software is clearly going to become an increasingly important part of the computer professional’s life. Consequently, we need to change the way we conduct the business of computers—or risk becoming part of an endless lose-lose litigation scenario. Actually, much of this change is good—good engineering practices, more reality, painful honesty with bosses and clients, and so forth— will benefit everyone in the long run. Finally, all software professionals must accept the obligation to always apply principles of worst-case engineering—in such common use by other engineering disciplines—or the legal profession will make it excruciatingly clear why the computer profession should have done so! References 1. T. DeMarco and T. Lister, “Both Sides Always Lose: Litigation of Software-Intensive Contracts,” CrossTalk, vol. 13, no. 2, Feb. 2000, pp. 4–6 ; www.stsc.hill.af.mil/Crosstalk/2000/ feb/demarco.asp. 2. W. Humphrey, Managing the Software Process, Addison Wesley, Reading, Mass., 1990, p. 59. 3. H.W. Lawson, “An Assessment Methodology for Safety Critical Systems,” contact the author at [email protected].

John Cosgrove, P.E., has been a software engineer for

over 40 years and a self-employed consultant for more than 30. His specialties include forensic engineering, project management, software architecture, real-time critical systems, and hardware– software interfaces. He has extensive experience with aviation computer systems, including development of aircraft navigation systems and communication devices. He has a Masters of Engineering from UCLA and a BSEE from Loyola Marymount, Los Angeles. He is a California-registered Professional Electrical Engineer, a senior member of the IEEE Computer Society, and a member of ACM and the National Society of Professional Engineers. Contact him at Cosgrove Computer Systems, Inc., 7411 Earldom Ave., Playa del Rey, CA 90293-8058; [email protected].

focus

guest editor’s introduction

Organizational Change

Ann Miller, University of Missouri–Rolla

alt Disney is credited with saying that “Change is inevitable; growth is optional.” We certainly have witnessed significant change in the software industry in the last few years. Furthermore, even the rate of change seems to be increasing. Will the pace level off or will it continue accelerating? And as we change, are we growing? Certainly the size of software in products is increasing. A few years ago, a study of one product family indicated that its software content was expanding at more than 80 percent

W

18

IEEE SOFTWARE

May/June 2001

0740-7459/01/$10.00 © 2001 IEEE

per year; furthermore, this product line had been experiencing that growth rate for nearly two decades.1 Are we growing proportionately in our knowledge of how to manage, develop, and maintain software? Lead, manage, cope, or none of the above? Organizational change is this issue’s focus. In the software and information technology industry, organizational change has been a way of life. It is quite telling to listen to individuals discussing change in their organizations. Their words frame their philosophies: some plan and lead change, others manage it, still others accommodate change, and many simply try to cope with it. Then there is the euphemistic phrase “change control.” I have always found this phrase particularly curious, especially since change control boards often don’t control change effectively. In a major system development project, I recall a situa-

tion in which a two-tiered hierarchy of CCBs was established: one senior-level CCB and several second-tier boards, one for each major subsystem. The chair of the senior CCB dictated that only the proposed changes whose impact was estimated to cost $1 million or more should reach this higher-level review and approval board. It was amazing how many proposed modifications were estimated to cost “only” half a million dollars. This dictate discouraged engineers from carefully considering and tracing the potential ramifications of their proposed changes. Furthermore, although representatives from other subsystem projects attended every meeting of the second-tier boards, they were not voting members. Thus, even if these codevelopers saw the potential for significant impact to their own subsystem from a change within a different subsystem, all they could do was voice that concern. To take the matter to the senior board, the affected subsystem group had to carry out its own study and demonstrate that the total cost of the proposed change would exceed the magic million-dollar mark. That project certainly did not control change effectively; on more than one occasion, senior management had to rescind an approved change after receiving a more detailed impact analysis. Change is not only inevitable, it is everywhere. Organizational change occurs at many levels and across many dimensions. When I worked in industry, my colleagues and I often commented on the organizational chart du jour; some part of our large company was always undergoing change. But change occurs not just in management structures; it also occurs in products, processes, technology, tools, communication media—in virtually every aspect of an organization, down to its very core: its corporate culture and people. Change usually also involves some element of risk. Conversely, when a potential risk becomes a reality, some sort of change becomes necessary. Some people thrive in the energy and turmoil of change; others resist it mightily. Thus, the articles appearing here address social issues as well as technical ones.

Some plan and lead change, others manage it, still others accommodate change, and many simply try to cope with it.

The theme articles Just as organizational change spans a wide spectrum, so too do the articles and features in this special focus. In the first article, Michael Deck addresses the importance of process diversity. In recent years, there has been a trend May/June 2001

IEEE SOFTWARE

19

People are always key to any process improvement, so methods to help staff ramp up on the learning curve of a technology or process are extremely important.

20

IEEE SOFTWARE

in companies to standardize the software development process; one supporting argument for this common process is that a software engineer from one project can easily move to a different project and be productive quickly in the new environment. However, many software engineers feel that one size does not fit all when it comes to development and test process. They frequently subvert or ignore a standard process because they feel it is too restrictive on a given project. “Managing Process Diversity While Improving Your Practices” addresses diversity management within an organization to tailor a standard process to meet specific project needs. We continue with software process improvement in “SPI Patterns: Learning from Experience” by Marina Blanco, Pedro Gutiérrez, and Giuseppe Satriani, who discuss the European Software Institute’s repository of over 250 process improvement experiments. The authors have analyzed the repository for patterns to help organizations plan improvement initiatives. In the third article, “Mentoring ObjectOriented Projects,” Ramkumar Ramaswamy discusses the value of on-the-job mentoring in learning process and design skills, particularly in object-oriented projects. People are always key to any process improvement, so methods to help staff ramp up on the learning curve of a technology or process are extremely important to the success of that technology or process adoption. The fourth article explores the business aspects of organizational change. In “What Makes Measuring Software So Hard?” Stan Rifkin discusses the importance in having a software measurement program aligned with the organization’s business goals and objectives. When working on software, requirements inevitably change and grow; moreover, each of the key stakeholders—customer, end user, manager, developer—approach the same requirements from different perspectives. The final article, “Developing Groupware for Requirements Negotiation: Lessons Learned” by Barry Boehm, Paul Grünbacher, and Robert O. Briggs, presents a distributed groupware system called WinWin and discusses how it facilitates the requirements process. In addition to the contributed articles, this issue features two interviews. The first is with Eric Schmidt, Novell’s chief executive

May/June 2001

officer. It is Novell’s latest ad campaign that triggered the interview: The use of David Bowie’s classic rock song, “Changes.” Schmidt has been at the helm of Novell through a sea of change, and he shares some of his insights gained over that time. When companies embark on a significant transformation, they frequently seek advice from consultants. Today, there is a new breed of consultants who prefer the title “coach.” They stress dialog, not one-way communication; they stress collaboration and teamwork; and they deal specifically with change. Our second interview is with Mary Boone, president of Boone Associates. She is an executive coach and consultant who specializes in organizational communication and the strategic application of information technology.

T

o bring some insight and stability in a time of change, Albert Einstein’s three rules of work offer good advice:

1. Out of clutter, find simplicity. 2. From discord, find harmony. 3. In the middle of difficulty lies opportunity. There are those who argue that change is just the swing of the pendulum; so perhaps the words of another philosopher, John Mellencamp, are appropriate when he sings, “I know there’s a balance—I see it when I swing past.” Acknowledgments Many thanks to the reviewers who contributed their valuable time and expertise in critiquing the articles that form this special focus.

Reference 1. T. DeMarco and A. Miller, “Managing Large Software Projects,” IEEE Software, vol. 13, no. 4, July 1996, pp. 24–27.

About the Author Ann Miller is the Cynthia Tang Missouri Distinguished Professor of Computer Engineering at the University of Missouri-Rolla and is the IEEE Software associate editor for software management. She also serves on the NATO Information Systems Technology Panel and chairs the NATO Task Group on Validation, Verification, and Certification of Embedded Systems. She has over 12 years of software experience in industry and three years of senior executive service in the US Department of Defense. She has a BS, MS, and PhD in mathematics from St. Louis University. Contact her at [email protected].

focus

organizational change

Managing Process Diversity While Improving Your Practices Michael Deck, Cleanroom Software Engineering, Inc.

Software process improvement efforts will fail if we try to make development processes completely uniform across large organizations. By focusing on localized software practice improvement, organizations can tailor processes to meet specific needs.

n recent years, the industry has gradually moved toward implementing standard processes across large organizations. The benefits of doing so include simplified accounting, measuring, and managing. Implementing organization-wide processes also makes it easier to judge the capabilities of an organization as a whole, which is important in largescale software development where organizational structures can have a profound impact on success.

I

0740-7459/01/$10.00 © 2001 IEEE

But there are also drawbacks to process uniformity, including cases where clean process documentation hides chaos in the trenches. Another drawback is the difficulty to modify an organization-wide process that has been carefully vetted by committee after committee. An organizationally uniform process can prevent smaller software teams working within large organizations from adopting a process tailored to fit their needs. I define software practice improvement as a set of goals that sits between individual programming practices and organizationlevel software process improvement. By focusing on improving software practices, we stand a better chance of achieving success using well-understood techniques such as risk analysis, incremental development, and team ownership. I define process diversity as using several different process variants simultaneously.

This article describes a project in which localized software practice improvements led to process diversity. It describes certain problems that we encountered and resolved during the effort. It also presents some simple techniques that any project—small or large—can use for managing diversity. The techniques I discuss in this article evolved in the course of one multiyear project. If you do not carefully plan and manage process diversity, it quickly becomes unmanageable. I used to advocate process uniformity, warning against the management problems caused by diversity, but recent experiences have suggested more effective ways to address these problems. This article explores these alternative strategies, which include adopting a basic set of practices that are uniformly enforced across all development steps, using risk analysis as a technique for selecting targets for software May/June 2001

IEEE SOFTWARE

21

If you do not carefully plan and manage process diversity, it quickly becomes unmanageable.

process improvement, and planning for process diversity. A software practice improvement story The project I explore here started with an enthusiastic team of six relatively inexperienced developers, plus a technical lead who had significant domain expertise from previous similar efforts. We had to develop a platform for real-time control of several optomechanical instruments. Several different missions were the customers for this platform, including two space missions and one ground-based observatory. We sized the project initially at approximately 70,000 lines of C and C++ code, most of which would run in the VxWorks operating environment.1 The project proved to be a good environment for trying out advanced software engineering techniques. With a small team, we didn’t expect management issues to swamp the technical issues. And, because the developers hadn’t learned to succeed by working within an existing flawed process, they were willing to try new ideas. The project’s developers had almost no software engineering training or experience, although they had been programming for some time and were domain experts in real-time controls, optics, and electronics. The distinction between programming and software engineering is intentional because the latter also includes elements such as requirements analysis, metrics, team organization, project estimating, planning, conducting reviews, and developing design documentation. We were determined to try new software processes and practices to avoid mistakes made on previous projects. However, we also planned to experiment with certain practices that the organization as a whole (a large government lab) had not yet approved. Risk analysis played a central role in defining our specific software practice improvement activities. Initial risk analysis We began our software practice improvement effort by analyzing risk. We used several sources to devise a set of typical project risks, but our main source was the Software Engineering Institute’s Taxonomy of Risks.2 We supplemented the SEI source with a riskguided analysis of several current quality

22

IEEE SOFTWARE

May/June 2001

programs, including the SEI’s Software Capability Maturity Model (CMM) and the Malcolm Baldrige award criteria. The assessment process included a questionnaire followed by meetings with the team. We determined that approximately 12 areas could be classified as high risk, nine as medium, and 11 as low. These numbers reflect the combination of over 100 individual risk criteria. Among the high-risk areas were several existing issues, including having ■ ■

no established process for requirements tracking and insufficient confidence in the requirements process.

There were also several additional potential risks, including ■ ■ ■

inconsistent planning for and documenting of project activities and commitments; uncontrolled change in processes; and instability of requirements, specifications, and designs.

We explored in detail some of the potential effects of these risks. For example, the lack of an established process for requirements tracking led previous projects to discover missed requirements late in the process. We were also concerned about requirements stability, because the project had many sponsors with very different needs. A single improvement path We developed an improvement plan to treat these risks in approximate descending order of importance and integrated the practice improvements into an incremental development model. Our idea was to deliver increments of product code that could be integrated piecemeal with the target hardware (also under development at the time). In each development increment, we would further improve and formalize the practices. Our increment model is similar to that used in the evolutionary (EVO) model,3 except that our increments were somewhat longer than EVO might recommend; they are on the order of three to six months each. The incremental-evolutionary model is key to our topic of diversity because such a model (as opposed to a waterfall approach)

permits controlled evolution of processes as well as products. The improvement plan set a high initial standard for software engineering practices. These practices included several techniques drawn from the Cleanroom software engineering approach:4,5 ■

■

■

formal behavior specifications using a notation based on the denotational semantics of Harlan Mills and his colleagues;6 reliance on team verification review rather than on testing for most requirements-coverage analysis; and numerous documents that were new to the project, including a detailed incremental project plan, user-view requirements, and formal specifications at the system, component, class, and method levels.

This plan was intended to apply uniformly to all project code, an aspect that did not seem significant at the time. We embarked on this learning process enthusiastically. Initial success The first increment’s goals were reasonably ambitious. The technical goals included proving the feasibility of numerous real-time control and scheduling algorithms, installing and learning a new compiler and runtime environment, and debugging a new optomechanical apparatus. Management goals included filling gaps in the domain knowledge of some team members, identifying significant gaps and inconsistencies in requirements, and identifying design trade-offs and alternatives. The first increment achieved all of these goals. The mandatory documentation and code reviews contributed greatly to crosstraining team members in domain knowledge. The formal specification with Box Structures uncovered numerous significant architectural and requirements problems that would likely have arisen much later in a traditional process. We found the documentation useful in focusing discussion on design trade-offs and alternatives. That the project also delivered significant functionality in the first increment demonstrated the capability of both the process and the team to meet critical user requirements. The first increment created an unex-

pectedly robust and maintainable architecture. Although little of the first increment remains in today’s product (due to a switch from C to C++), most of the architectural elements that were present in increment two remain in increment seven after almost two years of additional work. The steep learning curve did delay delivery of the first increment, though there were a number of unplanned outside events that also delayed the schedule. Perhaps most importantly, the team that developed the first increment was extremely enthusiastic about the Cleanroom process they had used and were committed to continuing its use.1

The mandatory documentation and code reviews contributed greatly to cross-training team members in domain knowledge.

Other risks emerge Unfortunately, the initial risk assessment did not account for all the actual risks that were to appear. Much of our inability to account for all the actual risks can be explained by pointing to changes in the project’s environment, but the fact remains that by increment two there were significant new risks. These risks did not fully manifest themselves until increments three and four, which is another reason why we did not address them quickly enough. The most significant change in the project’s environment was in the area of staffing. As the project grew, additional personnel joined the team. Most came from other projects that had used the organization’s traditional process. Many were more experienced, and more senior, than the original team. The arrival of the newcomers led to two kinds of problems. First, the original project team members had to train the newcomers in their practices before the newcomers could be productive. This lowered overall productivity at a time when schedule pressures were already high. Second, the senior transfers were comfortable with their traditional processes and were highly resistant to trying new techniques. Endless debates further reduced productivity. We also underestimated the impact of not having the team trained in basic software engineering principles. A team that had some training in the basics of inspections, requirements, and formal testing would have been more comfortable with verification-based inspections, formal requirements specifications, and stochastic testing. May/June 2001

IEEE SOFTWARE

23

In the absence of broad, grassroots acceptance of basic processes, proposed improvements will not survive the inevitable project stress events.

We also had to spend more and more energy explaining and defending our process to senior staff and upper management. We waged battles in email and hallway discussions. In the end, we could not enforce process compliance and still meet our aggressive delivery schedule. Resolution After an increment of partial anarchy, a consensus began to emerge around the possibility that a single software process and a single improvement path might be unrealistic for the project. In particular, we observed that there were really three kinds of activities going on at any one time. Some of the code was being built to support critical elements of space-borne scientific instruments. This code had to meet the highest level of quality and reliability but could be commensurately more expensive to build and document. Some of the code was being built to support noncritical components and ground-based instruments. This code did not have to meet the same reliability requirements as the critical software. Finally, there were a lot of necessary experiments to run. For example, would a particular memory-management strategy meet performance objectives? The simplest way to find out was to write a little code and run an experiment, but nobody wanted to subject the code to a formal process of specification, design, review, testing, and documentation. The result was a diverse set of processes in simultaneous use, each of which was tailored to meeting key needs of a software subset. Adopt basic practices Our experiences with this project led us to believe that no process, no matter how wellintended, can satisfy the needs of every project. Process diversity is, therefore, a fact of life. What we must do is manage that diversity. First and most importantly, we learned to define a set of foundation practices upon which there is near-universal agreement. Although we did do this in our project, we didn’t consider the pool of individuals who were likely to transfer into the project in the near term. Before embarking on a software practice improvement effort, the team must agree on—and ideally document—an initial set of universally accepted practices. If the initial

24

IEEE SOFTWARE

May/June 2001

set of practices isn’t broadly accepted, adding advanced practices will be more likely to fail. Consider the analogy of a house built in an earthquake zone. If the foundation is strong, the earthquake will just jar loose some trim. But if the foundation is weak, the entire structure will crumble. In the absence of broad, grassroots acceptance of basic processes, proposed improvements will not survive the inevitable project stress events such as turnover, reorganizations, requirements changes, and “crunch” mode (that painful time of extreme schedule pressure). The foundation practices must be part of the organization’s fabric so that people no longer question the practices’ worth or try to eliminate them in rounds of cost cutting. We recommend defining foundation practices in the following areas: ■ ■ ■ ■ ■ ■ ■

requirements management, software project planning, peer review, quality assurance, configuration management, metrics, and defect prevention.

Risk analysis will also guide the definition of further foundation practices, as will analysis of prior problems and crises. It can be challenging to define a set of consensus practices in some areas. It might require baby steps initially. For example, although defect prevention is an advanced practice of SEI’s Software CMM, most projects can find consensus to hold a regular informal discussion of significant software bugs, perhaps at the regular staff meeting. These sessions can start with an analysis of every bug that took more than two hours to fix. The discussion will naturally flow to how that bug or similar ones might have been prevented. After the meeting, it is likely that programmers will return to their desks and look for similar problems in their code. One of the questions not fully answered by the projects on which this experience report is based is who must participate in consensus-building. In these projects, we sought universal consensus only within our project, but that turned out to be insufficient because of personnel flow between the project and the rest of the organization.

Example one

On the other hand, seeking consensus and defining processes beyond a single project was perceived as being outside the project’s scope of responsibility. Indeed, the teams sought outside consulting advice because they were unsatisfied with weak organizational processes and sought to institute stronger local practices. Although educating the organization as a whole would have been useful, there were significant benefits to stealth, lest local decisions be overruled by the organization’s process engineers. Use risk analysis to select practices Once the foundation practices have become a comfortable part of everyday project operations, you can begin to identify targets for improvement. If you select the wrong targets for software process improvement, you will waste valuable resources fixing the wrong problems. For example, one project in a different organization was canceled because the team did a poor job prioritizing risk. The team spent months learning objectorientation so that their code would be extensible and maintainable, but their greater risk lay in not being the first product on the market. Understanding the threats makes it possible to avoid them. For this reason, our second diversitymanagement recommendation is to use risk analysis to identify process improvement opportunities. After agreeing on what the most serious risks are, software process improvements can be more focused, addressing the most significant problems rather than those that require the coolest tools. Risk analysis, prioritization, and process improvement must continue throughout the project, so that the process always tracks the current situation. Informal risk analysis means thinking about all of the ways projects have failed in the past and asking whether any of those factors are present in the current project. A risk taxonomy2 or risk assessment process7 can be easily tailored to your project. Armed with the risk profile, you then evaluate tools and techniques. The highest risks will be expensive to address, so they must be subdivided into manageable units. Figure 1 shows two examples of how it can be done. Risk analysis is by no means the only way to identify software practice improve-

Past situation: A new project member needed weeks of training and code-reading before being able to make even a small change. General risk areas: documentation, design, and training. Specific risks: The current design documentation is organized to be more useful as a reference than it is for learning. Possible near-term practice improvements: Create road map and survey documents that don’t duplicate content but serve as guides for learning. Produce examples of typical tasks. Investigate documentation-management tools that support version control. Success criteria: Significant decrease in amount of time between joining the team and making useful contributions.

Example two Past situation: You communicated requirements changes to a junior staff member without discussing the changes with the team; the implementation led to errors. General risk areas: Requirements stability, requirements traceability, design communication. Specific risks: Requirements changes are accepted without project review. Possible near-term practice improvements: Create a simple database or spreadsheet that lists all requirements, even if vaguely stated. Convince requirements-givers that every change must be logged, then use reviews to approve updates to database. Success criteria: All requirements changes are logged for review and analysis.

Figure 1. Two exam-

ment targets, but it does lend itself to a ples of subdividing greater or lesser degree of rigor. It is also into manageable one that few small teams think to apply, units even though it is common practice at the level of large projects and organizations. Used informally at the team level, it can be a valuable guide to practice selection. Plan for diversity The only way to avoid process diversity is to adopt the perfect process and then never change it. Failing that, the rest of us will be trying out practice improvements on a pilot project or component while leaving other parts of the project or organization alone. Some of the practice improvements selected through risk analysis might become part of a revised foundation process; others will be applied in special circumstances, and yet others will be rejected as inappropriate or unsuccessful. One approach is to segment your processes based on the software quality factors or “ilities” such as capability, reliability, and usability. The project I described here segmented its processes based on reliability: the flight software had higher reliability requirements so it needed a tougher process. You May/June 2001

IEEE SOFTWARE

25

Table 1 Multitier Process Guide Description 1 Use CVS as repository of all code

2 3 4 5 6

Each module must have conforming header comment Use formal ‘B’ language specifications for every method Conduct ‘buddy’ review of all code created or changed Conduct formal inspection of all code created or changed Track all labor hours spent in review/inspection

Foundation process Required

Standard process Required

Advanced process Required

Required Optional Optional Optional Optional

Required Optional Required Optional Recommended

Required Required Recommended Required Required

could also base your decisions on something like usability: certain projects will have mandatory usability testing, while others will not. Sill another approach would be to focus on maintainability: certain projects will produce code that is used and maintained over a long time; others produce one-off, shortterm solutions. This is not to say that singling out one software attribute is always the right way to segment processes. The important thing is to consider the impact of each process or technique on all of the software quality factors. A technique that has worked for us is to define all of the possible process attributes but then identify tiers or strata that define different practice groups for different types of code. Table 1 shows a segment from a multitier process guide. Table 1 indicates that every code unit must be checked into the CVS repository. It is universally acknowledged that CVS will be the repository and not some other product. Furthermore, everyone agrees that each module’s header comment will conform to a particular standard. We have also decided that buddy reviews are a technique that might be useful on some projects, so we list that in our process database. Then we determine that the foundation process should consider that technique to be optional, while the standard process should always use that technique. The advanced process will also be required to conduct formal inspection, but pre-inspection buddy reviews are recommended. The process tiers are not strict subsets or supersets of each other—numbers 2 and 3 in Table 1 define variants of the same practice area. In the project described here, we applied the foundation process to one-off experiments and simple feasibility studies, whereas we applied the standard process to noncritical software and the advanced process to critical sections and controls. 26

IEEE SOFTWARE

May/June 2001

There are two important attributes to managing diversity within the parameters of this kind of plan once you have defined each process’s and each segment’s required, recommended, and optional elements. First, you must clearly define which products will use which processes—and who will decide. If you do not clearly define this in advance, crunch mode will reduce every project to the least-strict process. Second, you must have a mechanism through which one process’s work products can be used by another process. For example, code developed as part of a one-off experiment should not be used in the spaceborne software kernel without some additional reviews and testing. However, we shouldn’t mandate that all experiment code be thrown away either. The right thing to do is to treat code developed under a less-strict process as raw material that must be documented, reviewed, and tested within the more-strict process if it is to be incorporated into a component that requires it. Enforcing this requirement is a key element of day-today technical management. Unfortunately, moving work products from one process to another is not always a black-and-white activity, and its management is not always easy. We found numerous examples of work that drifted across the boundary, mostly through code reuse by engineers.

P

rocess diversity adds complexity to the search for standards compliance with models such as CMM and ISO. In general, a company’s CMM level will probably be defined by the common subset of all of its processes, which is unfortunately the weakest process level. This points to both a problem with the diversity approach (possibly undermining organizationwide SPA efforts) and with the current SPA

About the Author Michael Deck is a software practice consultant at Cleanroom Software Engineering,

frameworks (which penalize organizations for locally improved processes). Organizations that seek higher CMM levels will need to upgrade the minimal process level continually to include new areas of consensus. When seeking CMM compliance, you must evaluate all processes against the relevant key process areas. In the case of ISO, each process must be well defined within the ISO framework with the appropriate documents and records. Each project team member must be aware of which process they are using for each work effort. For a project to make long-term progress toward improved quality and productivity, all team members must agree on a set of basic engineering practices and use risk analysis to identify advanced techniques. The resulting diversity must be carefully managed to prevent chaos and backsliding. Acknowledgments I thank Brad Hines of the Jet Propulsion Laboratory for his help and encouragement, and the anonymous reviewers for suggesting many content and presentation improvements.

Inc. His research interests include applying Cleanroom models to object-oriented development, real-time systems, and embedded software. He specializes in training project teams to tailor and use software engineering practices to solve real-life software process problems. He received an MS in Computer Science from the University of Maryland, College Park. Contact him at [email protected].

References 1. M. Deck and B.E. Hines, “Cleanroom Software Engineering for Flight Systems: A Preliminary Report,” Proc. IEEE Aerospace Conf., IEEE Press, Piscataway, N.J., 1997. 2. M.J. Carr et al., Taxonomy-Based Risk Identification, Tech. Report CMU/SEI-93-TR-6, Carnegie Mellon Univ., Pittsburgh, June 1993. 3. T. Gilb, Principles of Software Engineering, AddisonWesley, Reading, Mass., 1988. 4. S.J. Prowell et al., Cleanroom Software Engineering: Technology and Processes, Addison-Wesley, Reading, Mass., 1999. 5. M.D. Deck, “Cleanroom Software Engineering: Quality Improvement and Cost Reduction,” Proc. Pacific Northwest Software Quality Conf., 1994, www.cleansoft.com. 6. R.C. Linger, H.D. Mills, and B.I. Witt, Structured Programming: Theory and Practice, Addison-Wesley, Reading, Mass., 1979. 7. K. Wiegers, “Know Your Enemy: Software Risk Management,” www.processimpact.com/articles/risk_mgmt.

Introducing the IEEE Computer Society

Career Service Center Career Service Center • Certification • Educational Activities • Career Information • Career Resources • Student Activities • Activities Board

Advance your career Search for jobs Post a resume List a job opportunity Post your company’s profile Link to career services

computer.org/careers/

computer.org

May/June 2001

IEEE SOFTWARE

27

focus

organizational change

SPI Patterns: Learning from Experience Marina Blanco, Pedro Gutiérrez, and Giuseppe Satriani, European Software Institute

The European Software Institute maintains one of the most important This

housands of organizations have been improving their software process maturity level since the Capability Maturity Model (CMM) appeared in 1991. Organizations have improved project management, automated configuration management, and introduced reuse practices. Some of them succeeded; others failed. If we could extract lessons for the software community from these experiences, organizations could avoid repeating mistakes.

T This article describes a repository of 400 process improvement experiments and presents patterns that can help organizations plan their improvement initiatives. 28

IEEE SOFTWARE

Publications such as the Process Maturity Profile of the Software Community1 (published yearly by the Software Engineering Institute) show organizational trends assessed against the SW-CMM. They provide information about the organizations’ state and the time spent moving from one level to the other, but say nothing about the different ways used to reach that state. Unfortunately, such information is embedded in experiences that are not public. However, initiatives such as the European Systems and Software Initiative facilitate some of these experiences. The European Commission has been funding around 400 process improvement experiments (PIEs) through ESSI since 1993. More than 250 are public and available through Vasie, a repository that can be accessed through the Internet. Many researchers have analyzed this repository (and the ESSI) many times2–4— May/June 2001

but none have focused on identifying patterns of solutions for specific problems or patterns for pursuing concrete business goals, or on determining that pattern’s relative success of such patterns. From architecture to software design, many disciplines have identified patterns by the analysis of past experiences5–6—We believe this approach is also valid for software process improvement. Vasie The European Commission, aware of software’s increasing technical, strategic, and social importance, created the ESSI to improve European industry competitiveness by improving software process capability. This program pushes software process improvement through the direct funding of PIEs and the dissemination of the achieved results. The Vasie project is a dissemination 0740-7459/01/$10.00 © 2001 IEEE

action, started in 1993 to collect, validate and disseminate the PIEs’ results. To achieve this dissemination objective, the European Software Institute has developed a public repository containing the PIEs’ results. Since 1995, Vasie has been continuously receiving data; today (Jan. 2001) the repository contains the experience of 273 organizations. Vasie is continually growing and can be accessed through the ESI Web site (www.esi.es/vasie). Vasie is the most important and largest repository of SPI experiences and is representative of the different European industries. It was designed so that organizations could learn how others had designed improvement projects, the problems they encountered, the solutions they devised, and the context in which the experiments had been performed. A large number of organizations download around 1,300 reports from Vasie every month. This figure indicates the acceptance of Vasie as an active instrument for designing SPI projects, for supporting consultants activity, and for extracting case studies for classroom discussion. What is a PIE? A PIE (see Figure 1) is a project that aims to demonstrate the benefits of improving the software development process through a controlled, limited experiment. It consists of five phases: 1. Identify the improvement needs of the organizations that participate in the PIE. 2. Develop solutions that help address the improvement needs. 3. Try the solutions proposed in one or more software projects of the participating organizations. These software projects are called baseline projects in ESSI terminology. In a baseline project, an organization tries out new procedures, new technologies, and organizational changes before deciding whether or not to replicate them throughout the software-developing unit. 4. Evaluate the impact of the solutions. 5. Deploy successful solutions to the rest of the projects in the organization.

Dissemination

PIE Analysis of starting scenario

Experimentation

Analysis of resulting scenario

Next stage

Baseline project

Figure 1. A process improvement experiment.

Vasie content overview In January 2001, Vasie contained 246 PIEs; because more than one organization can participate in each PIE,7 about 273 organizations were represented. The analysis presented in this article was performed in September 1999. At that time, Vasie contained 184 PIEs and 216 organizations. More than 11 countries are represented in Vasie. This sample of SPI experiences is one of the largest ever analyzed, so even if it is not considered representative, it is relevant and provides one view of the state of European SPI. The complete results of this study are presented elsewhere.2 The PIEs were executed in software development units of various sizes. Units of fewer than 20 employees carried out 37 percent of the experiments; groups with 20 to 60 employees, 27 percent; units between 60 and 150 employees, 23 percent; and groups with 150 to 10,000 employees, 13 percent of the experiments. The three most frequently pursued business goals were improved delivered quality, cost re0 months

12 to 18 months

24 to 30 months

PIE Starting Scenario Questionnaire

Midterm Report

1 year Final Report and Resulting Scenario Questionnaire

Impact Analysis Questionnaire

Vasie

Most PIEs have sent information to Vasie following the flow shown in Figure 2. Figure 2. Information flow from the PIE to Vasie.

May/June 2001

IEEE SOFTWARE

29

Group A Maturity level

Group B

Level Y 13%

Level Z 11%

Level X 87%

Primary involvement in software industry

Software user (primarily developed by a third party) 11%

Baseline project characteristics

Level Y 67% Software vendor (producing custom software systems) 15%

Software user (primarily developed in-house) 89%

Level X 22%

Software vendor (producing off-the-shelf systems) 23%

Software user (primarily developed by a third party) 8%

Software user (primarily developed in-house) 54%

Production 13% Laboratory 33%

Production 67% Laboratory 87%

Industrial sectors

Textile and textile products 13% Software consultancy and supply 13% Machinery, electrical, and optical instruments 24%

ISO 9001

Agriculture and forestry 13% Electronic components 24%

Telecommunication products 13%

Electrical engineering 11% Mechanical engineering 11% Telecommunication products 11% Finance and insurance 11%

Certified 13%

Machinery, electrical, and optical instruments 23%

Software consultancy and supply 22%

Health 11%

Not certified 11%

In the certification process 13% Certified 56%

Unknown 74% Key

Level X = lack or ineffective use of project management

Table 1. Characteristics of Groups A and B.

30

IEEE SOFTWARE

Unknown 33% Level Y = project management function exixts at project level

duction, and increased productivity. The three most frequently tackled problems in order of importance were reusability, project cost devi-

May/June 2001

Level Z = software projects use organizational procedures and methods

ation, and lack of process definition. Vasie recognizes three categories of capability levels. Level X indicates the lack or in-

effective use of project management. Level Y means that a project management function exists, at least at the project level. Level Z means that software projects use organizational procedures and methods according to a software development life cycle defined at the organizational level. Table 1 offers an informal mapping of organizations and levels. The PIEs addressed these improvement areas (process categories according to ISO/IEC TR2 15504 1998): ■ ■ ■ ■ ■

engineering (65 percent), support (14 percent), management (8 percent), organization (8 percent), and customer–supplier (4 percent).

Patterns Analysis of the data in the Vasie repository reveals patterns of solutions that are commonly applied by many organizations. A pattern is an insight that conveys a proven solution to a recurring problem within a certain context amid competing concerns. To identify patterns, we follow these steps: 1. Identify the objective or problem for which patterns are to be identified. 2. From the repository, obtain the PIEs that address the identified objective or problem. 3. For each PIE, retrieve the set of processes affected. 4. Whenever the intersection among these sets is not empty, a pattern is found. The patterns we identified have three components: the problem to be solved (or the goal pursued), the context where the solution is applied, and the solution applied, represented by the set of improved software processes.8 There is a fourth component, though, the number of organizations where the pattern has been identified (although this fourth component is not actually part of the pattern). According to this pattern representation, two patterns are different if they differ in one or more components. A pattern is usually represented as a picture. The different patterns that tackle the same problem or pursue the same goal are represented in the same picture (Figure 3). The problem to be solved, or goal, is signified by the big circle at the top of the picture. At the bottom of the picture, polygo-

The Data Collection Process The PIE sends the Starting Scenario Questionnaire to the participating organizations asking about the organization context, the objectives pursued, and the problems the organization wants to tackle with the experiment. This and the Resulting Scenario Questionnaire were designed on the basis of Vic Basili’s ideas.1 The mid-term report reflects the activities performed by the PIE up to the middle of the experiment. A PIE usually lasts from 12 to 18 months. Once it is finished, a final report of the experiment is submitted, summarizing problems, solutions, implementation steps, difficulties found, benefits achieved, and lessons learned. Submitted with the final report, findings from a Resulting Scenario Questionnaire reflect the processes changed, the facilitator, and the inhibitors encountered. Approximately one year after the PIE ends, an Impact Analysis Questionnaire is sent to the original participants. Those findings, reflecting the longterm benefits achieved and follow-up activities started, are submitted. Every PIE goes through a review process to decide whether the final report contains relevant information, reproducible solutions, and valuable lessons and benefits. Those experiments that fulfill the review process criteria go into the repository for public dissemination.2

References 1. V.R. Basili, R.W. Selby, and D.H. Hutchens, “Experimentation in Software Engineering,” IEEE Trans. Software Eng., vol. SE-12, no. 7, July 1986, pp. 733–743. 2. Impact Analysis: Initial Results, FZI Research Center for Information Technologies, Univ. of Karlsruhe, Karlsruhe, Germany, 2000.

nal figures each represent a set of organizations where the pattern has been identified (the bigger the polygonal figure, the higher the number of organizations that follow the pattern). The context of application, that is, the characteristics of each group of organizations, is represented in Table 1. The solution is denoted by a set of circles, each representing a process or a group of processes. Finally, the lines from the bottom to the top of the figure, crossing the different processes, are commonly called roads. A road’s width represents the number of organizations that improved the process (the road “traffic”). Here are tips to better understand a pattern picture: ■

■ ■

The order in which a road crosses through processes does not necessarily indicate the order in which the organizations improved the processes. The road length at a given position does not indicate the effort involved. We’ve omitted processes with very low traffic for simplicity’s sake. May/June 2001

IEEE SOFTWARE

31

Increased delivered quality Verification

Project management

Software requirements

System requirements

Software design Group A

Figure 3. Two different patterns to increase delivered quality.

32

IEEE SOFTWARE

System and software integration and testing Group B

tions in that group decided to improve that process. From this process on, three new roads emerge. The left road goes directly to the goal. This means that a subgroup of Group B only improved software and system integration and testing. The middle road crosses through the verification process, and the right road crosses through project management. We can interpret Group A in a similar way. Table 1 shows the context in which these two groups of organizations implemented the solutions. The pie charts in the table measure these characteristics: ■ ■

■

Pattern example We collected all these parameters (objectives, problems, processes, context) from organizations through questionnaires (see “The Data Collection Process” sidebar). Figure 3 shows various patterns being derived from the Vasie repository, based on the experiences of 18 organizations. These organizations shared the goal of increasing the delivered quality of their software products. The pattern highlights the solutions they used to reach that goal. The pattern shows solutions applied to reach the goal rather than solutions that effectively helped to achieve the goal, because the pattern was identified before the impact analysis results were available. We have identified two patterns. The first, found in nine organizations, we call Group A; it consists of software design, system requirements, and software requirements. The second pattern, also identified in nine different organizations, is Group B; it consists of system and software integration and testing, project management, and verification. According to Figure 3, Group A focuses on engineering processes, especially on the cycle’s initial processes. However, Group B goes beyond engineering; it improves processes such as project management and verification. Only one road travels from Group B to the integration and testing of software and systems. That means that all nine organiza-

May/June 2001

■ ■

Maturity level. Primary involvement in the software industry, that is, the role of the organizations in the software industry. Baseline project characteristics. Remember that a PIE is implemented in conjunction with a baseline project. A project is a laboratory project when it is an off-line environment that resembles the actual software development environment. A project is a production project when its primary objective is to deliver a software product to a customer. The industrial sector to which the organizations belong. Whether the organization is ISO 9001 certified.

Table 1 data suggests that organizations at Level Y and organizations that are ISO 9001 certified prefer Group B’s solution, while Level X organizations and uncertified organizations prefer Group A’s solution. Another important characteristic about each pattern’s reliability level is that Group B’s pattern is tested in more realistic baseline projects, such as production projects. You could therefore expect that the results of that solution are more realistic as well. However, it could be interesting to understand whether the solution helped the organizations reach the goal. It could also be interesting to know the average effort for each solution’s implementation. This and other information are crucial for deciding which solution to implement; without them, a pattern is not as useful. This situation is resolved with the analysis of the Impact Analysis Questionnaire.

Group A Maturity level

Group D2 Level X

Level Z

Level Y

Primary involvement in software industry

System developer (develops most of the parts of the system)

Interest group (for example professional society or standards body)

Software vendor (producing custom software systems)

Software vendor (producing off-the-shelf systems)

Software user (primarily developed in-house)

Research and development (institute or university) Software vendor (producing custom software systems)

Baseline project characteristics

Production

Laboratory

Laboratory

Industrial sectors

Fabricated metal products, except machinery and equipment Electric components Aircraft and spacecraft Health and social work Telecom products

Software consultancy and supply, data processing, and related services

Production Finance and insurance

Technical testing and analysis Machinery, electrical, and optical instruments

Software consultancy and supply, data processing, and related services

Industrial process control

ISO 9001 Certified

Certified

Not certified Unknown

Impact analysis results The Impact Analysis Questionnaire (see “The Data Collection Process” sidebar) was designed based on the idea that most of the

Not certified

Unknown

In progress

Table 2. CharacterisPIE benefits would become evident only in tics of Groups A and the long term. It collects information from D2.

the organizations at least one year after the PIE is finished, making it possible to see May/June 2001

IEEE SOFTWARE

33

Figure 4. Patterns after the impact analysis results.

Increased delivered quality A2

B1

C2

C1

Implement Project Measurement software management design

A1

Organizations Group C

1. Only 19 organizations performed a follow-up action. 2. The achievement of technical objectives is higher than the achievement of business goals. 3. Fifty-seven organizations answered that the PIE’s original goals were too ambitious. Only 24 answered that the goals were correctly chosen. 4. The organizations saw little evidence of quantitative benefits. 5. Regarding the subjective evaluation of business benefits achieved, most respondents agreed that the only benefit was an increase in the product’s quality. The PIE did not affect cost reduction, staff productivity, time to market, or market benefits. May/June 2001

Configuration management

B2.2

Software integration

whether or not the organizations succeeded in reaching their goal. The questionnaire consisted of more than 60 questions inspired by the Software Engineering Institute’s style.9 The questions were grouped in three sections. The first section centered around the main elements that characterized the diffusion and institutionalization of the improvement throughout the organization. The second section ascertained whether the organization considered the PIE worthwhile and determined the main lessons learned. The third section focused on the benefits achieved, both tangible and intangible. On the basis of the 91 questionnaires we received, we reached these main conclusions:

IEEE SOFTWARE

D1

Software testing

System analysis analysis design Organizations Group A

B3

Integrate & test systems

Software design

34

D2

B2.1

Organizations Group B

Organizations Group D

We present an overview of the initial analysis of results elsewhere.10 Once the initial results of the impact analysis were ready, another statistical analysis was performed to identify the patterns most successful at increasing the delivered quality (see Figure 4).11 Group D reported the highest success improving both configuration management and project management (Road D2). Three organizations followed this road, and all of them exhibited evidence of maturity level Z. Group A reported the second most successful road, and Group B reported the third most successful road. Organizations in Group B improved software integration, software testing, and system integration and testing of Road B2.2. All the organizations in Group B were at maturity level X.

T

he most successful pattern found in Vasie, to increase the delivered quality of software products, includes two key process areas of CMM v1.1 at Level 2. Software vendors whose maturity level was already close to Level 2 reported this success (see Table 2). However, the sample behind the pattern (three organizations) is not significant. Publications such as the SEI Process Maturity Profile which is based on more than 1,000 organizations, might make you think

that CMM drives all the improvement efforts of the software community.1 We believe that this is not the case and that alternative improvement paths applicable in specific contexts exist but their benefits have not been confirmed. Measuring a PIE’s benefits is not easy if it was not designed to quantify them. Our hypothesis is that benefits would be more visible in the long term, which is why we designed the Impact Analysis Questionnaire. However, if measuring the benefits is not a PIE objective, making them visible will be difficult, even in the long term. Other types of patterns can be found in Vasie. This article explores one specific way of analyzing the repository; many others are possible. We encourage further analysis of this and other repositories and recommend Vasie as a source of valuable examples of SPI initiatives.

References 1. Process Maturity Profile of the Software Community 1999 Year End Update, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 2000. 2. M. Blanco et al., Repository of Expertise Guide, tech. report ESI-1999-TR-017, European Software Inst., Bizkaia, Spain, 1999. 3. G.W. George, Software Best Practice Experiences of the ESSI Pilot Phase Experiments, tech. report, European Software Inst., Bizkaia, Spain, 1998; www.esi.es/ESSI/ Reports/All/10807/Download.html. 4. 1995 Software Best Practice Questionnaire: Analysis of Results, tech. report ESI-PIA-960464V5, European Software Inst., Bizkaia, Spain, 1996. 5. C. Alexander et al., A Pattern Language: Towns, Buildings, Construction, Oxford Univ. Press, New York, 1977. 6. E. Gamma et al., Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley, Reading, Mass., 1994. 7. V.R. Basili, R.W. Selby, and D.H. Hutchens, “Experimentation in Software Engineering,” IEEE Trans. Software Eng., vol. SE-12, no. 7, July 1986, pp.733–743. 8. Software Process Assessment: Parts 1-9, tech. report ISO/IEC TR-2 15504, Int’l Organization for Standardization, Geneva, 1998. 9. J. Herbsleb et al., Benefits of CMM-Based Software Process Improvement: Initial Results, tech. report CMU/SEI-94-TR-13, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 1994. 10. Impact Analysis: Initial Results, FZI Research Center for Information Technologies, Univ. of Karlsruhe, Karlsruhe, Germany, 2000. 11. B. Kitchenham, “A Procedure for Analyzing Unbalanced Datasets,” IEEE Trans. Software Eng., vol. 24, no. 4, Apr. 1998, pp. 278—300.

About the Authors Giuseppe Satriani is manager of the

Strategic Software Process Improvement Unit at the European Software Institute. He is responsible for the development of ESI technologies related to improvement and management models and methodologies, such as CMM, SPICE, EFQM, Balanced Scorecards, and ISO 9001. He has been involved in many European projects, including Vasie (ESSI 24119), Aesopian (TE-2001), Riskdriver (Esprit 27890), and Patterns (IST-2000-25120). He is a SPICE assessor, an EFQM evaluator, an SEI qualified instructor for CMM, and an SEI authorized supervisor of CMM instructors. He received his BS in computer science from Pisa University. Contact him at Parque Tecnológico 204, E-48170 Zamudio, Spain; [email protected].

Pedro Gutierrez Negro is a consultant

in the Strategy and Software Process department at Sociedad para la Innovación Tecnológica. Previously, he worked at the European Software Institute, where he specialized in software process assessment, software process improvement, and software best practices. He received a BS in computer science and an MS in software engineering from Deusto University. Contact him at Olatxu, 5, E-48499 Zeberio, Spain; [email protected].

Marina Blanco Escudero is a consultant in the Strategy and Software Process department at Sociedad para la Innovación Tecnológica in Bilbao, Spain. Her main activities are the definition, maintenance, and improvement of the quality system (ISO-9001 certified) of a major Spanish energy production and distribution company. She previously worked as a software process engineer at the European Software Institute, where she actively worked in software process assessment and improvement and experience repositories. She received her BS in computer science from Deusto University. Contact her at Calle Zamakola, 37, E-48003 Bilbao - Vizcaya, Spain; [email protected].

May/June 2001

IEEE SOFTWARE

35

focus

organizational change

Mentoring Object-Oriented Projects Ramkumar Ramaswamy, Infosys Technologies

n mentoring, an experienced person (the mentor) undertakes to guide a less experienced person (the “mentee”) in the same or similar field. Mentoring can effectively bring an organization up to speed with a new technology.1 What loosely distinguishes mentoring from training is that the latter is something often associated with a classroom (we talk of “classroom training” but not of “classroom mentoring”). With mentoring, the learning happens on the job: a mentor educates by continuously hand-holding

I The mentor must be architect, design consultant, and educator on process and language issues. This multifaceted role presents unique challenges. 36

IEEE SOFTWARE

the mentee on the latter’s tasks. Of course, in any real situation the education process must involve a judicious and inseparable mix of training and mentoring, with the same person often performing both roles. You can apply mentoring to people in your organization or even in a customer organization; this article focuses on the former. Mentors are particularly crucial for an organization learning object orientation.2 (For a comparison of mentoring for OO and other types of projects, especially projects that use traditional Structured Development methods, see the “What’s the Difference?” sidebar.) In this article, I present lessons learned from my experiences as a mentor on OO application development projects. I have attempted to generalize my observations and present them as succinct, shop-usable advice—not just for the mentor but for the mentee as well. My experience Over the last few years, I’ve been a mentor on a number of OO application devel-

May/June 2001

opment projects. My involvement varied from being a full-time participant in requirements specification and design to being an “offline” or part-time consultant. These projects varied significantly. The earliest was an internal pilot that was instituted with OO mentoring as an explicit objective. It involved redesign and coding of a piece of custom-built middleware, spanned five months, and involved six people at its peak. This pilot “failed” in that we never actually deployed its deliverables; nevertheless, it achieved some degree of mentoring success, although not as significant as with later, live (customer) projects. An early live project was a 25person-month GUI-intensive pilot for a large utility firm; this project’s design philosophy drove several subsequent reengineering and development projects over three years. A more recent experience was with a 120-person-month project for a leading credit card firm, during which we developed a high-volume, high-availability back-end process (that is, with no user in0740-7459/01/$10.00 © 2001 IEEE

What’s the Difference? How is mentoring for OO projects similar to or different from mentoring in general or with more traditional methodologies? The table below provides observations on six key mentoring tasks. Many tasks differ in the OO context only in their intensity. However, certain tasks, such as getting the design done, might require a significantly different approach—sometimes bordering on the radical.

Table A Advice on Key Tasks for Mentors Mentoring task

General lessons

OO-specific lessons

Choosing or defining a methodology

The choice or definition need not be based on hype, notaation, or even tool support. Focus on a minimal set of modeling techniques appropriate for the context and mentees’ skill level. You can teach requirements modeling by itself, without language education as a prerequisite. Also, demonstrating the usefulness of models such as dataflow diagrams or the need for rigor is tough, owing to lack of code traceability. This is important but might not always be critical to maintainability. Mediocre procedural code can be an irritant but is maintainable as long as it is reasonably granular and not monolithic. The mentor can have mentees learn by doing, if elements of mediocre or immature design will unlikely do much harm. The mentor’s role then involves more critical reviewing.

Judicious use of Structured Development techniques and principles can ease the transition to OO. This might be politically difficult, so be prepared to demonstrate technical correctness. Greater traceability means that developers will more greatly appreciate the usefulness of, and the need for rigor in, object modeling if you provide language education as a prerequisite. This is critical. Mediocre OO designs might lead to downright unmaintainable code. However, good OO programmers don’t necessarily understand good OO design. Except with internal pilots, design criticality (see above) requires the mentor to do most of the architectural design and let first-time OO designers learn by watching, not doing. The emphasis on iterative development, the denouncement of the waterfall model, and the higher level of traceability lead developers to believe, incorrectly, that this separation is no longer as important. Reuse is the single highest expectation, often unfortunately at the code level. Worse, developers often expect reuse from artifacts as granular as individual classes. You must deflate this expectation.

Teaching requirements modeling

Imparting design education and making mentees aware of its importance Getting the design done

Separating analysis, design, and implementation concerns

Managing expectations of reuse

You must emphasize this. It is particularly easy with Structured Development techniques: the waterfall model, which is usually associated with these techniques, emphasizes the importance of this separation. Developers do not expect high reuse from traditional methodologies. You must mentor and encourage developers to design common functions that might be usable beyond the current application’s scope.

terface) for credit card transaction capture and processing. One factor that has considerably influenced the nature and intensity of my mentoring is a perennial labor shortage. The number of our developers who are even moderately familiar with the OO life cycle is far fewer than the number of developers we need on OO projects at any point in time. So, much of the learning must take place during live projects, leaving little room for learning through failures. The focus of mentoring The key technical areas that require training or mentoring are process, design, and language. I have found that traditional classroom-and-lab training effectively builds language and platform skills. However, process and design—particularly the latter—require mentoring, so I focus on these areas here. The amount of literature (and controversy!) on these two areas is overwhelming; the

mentor has the responsibility to assess and evaluate the state of the art in the context of his or her organization. Interestingly, some mentoring in requirements modeling might also be required, because developers familiar with traditional requirements-modeling techniques are often unsure of the role these techniques play in the OO context. A nontechnical area that generally requires mentoring is project management. However, given OO’s significant learning curve, only experienced managers (even if they are new to OO) should manage firsttime OO projects. This has been the case for all the projects with which I have worked. This strategy limits mentoring’s scope (and the associated risks) largely to technical issues. The project manager herself might opt for traditional training to develop an awareness of issues peculiar to an OO project, rather than aim for technical proficiency through mentoring. In this article, I focus on the issues that arise in the mentoring role May/June 2001

IEEE SOFTWARE

37

It pays to take the stand that OO programming skill is not a prerequisite for OO design skill.

38

IEEE SOFTWARE

per se, rather than on OO or project management issues. Managing team perceptions Although most developers are delighted to participate in an OO project, they seldom view an OO Analysis and Design method as anything more than a documentation technique. What benefits should you project from an OOAD method? Developers typically want and expect code-level reuse. However, in my experience, you’d be wise to deflate expectations of reuse in any custom-built software. Instead, you should promise greater maintainability and adaptability through increased modularity. Additionally, use cases as presented in books usually look “just like a good old functional specification” to a developer with some requirements specification experience in a traditional context. I have found it politically correct to downplay use cases. Instead, I promote object modeling as the common thread tying requirements, program design, database design, and code. As I highlight later, the value added by the mentor will probably peak during the object design phase. OO design is a skill in its own right, one that is not necessarily built on a layer of analysis or OO programming skills. (Luke Hohman gives an alternate viewpoint.3) Most developers appreciate that this skill is different from that of an analyst; surprisingly, however, they do not recognize that it is significantly different from the skills of a good OO programmer. Mentors need to keep in their armory a pack of “design posers” that they use to gradually create this recognition. These can range from the simple “How would you create the class stack by reusing the code for the class list?” to the more complex “Create an object model that represents a developer who can act as trainee, project leader, and database administrator in any arbitrary combination.” (The first poser leads to a discussion of subtypes and the Liskov Substitution Principle, which is a rule for assessing an inheritance hierarchy’s robustness. The second addresses the tradeoffs in using aggregation instead of multiple inheritance to model multiple roles.) Is fluency in an OO programming language necessary for a mentor to stand his ground amongst a battery of skilled OO pro-

May/June 2001

grammers? My experience indicates it isn’t. Pleading intentional ignorance on languagespecific issues and artifacts such as the size of the virtual function table or friends in C++ prevents language-specific issues from sidetracking design discussions. The prior use of design posers can enhance this tactic’s efficacy. In several (but not all) cases, I have been able to successfully convey to the team that OO programming skill is not a prerequisite for OO design skill and that fondness for a particular language might actually distort your approach to OO design. Of course, the mentor must be able to address language experts’ objections to a given design and resolve them suitably. For example, programmers generally avoid having two classes point to each other (sharing a bidirectional association). However, an instance of this in one design simplified object collaborations enough to warrant its inclusion. Reflections. Advocate modularity but downplay reusability. It pays to take the stand that OO programming skill is not a prerequisite for OO design skill. Build a repository of language-independent “design posers” to justify this stand. Negotiating the learning curve OO’s steep learning curve and the associated training issues have received much attention.4 To teach OO, should you use a comprehensive case study, or should you use assignments that focus separately on language, process, and design? In the projects with which I was involved, developers were already proficient in the language (usually C++), so we focused on process and design. When we used case studies, people strongly tended to mix up issues. For instance, the correctness of the inheritance notation being used tended to distract from the design issues involved in, say, mixed-role cohesion (a term that characterizes the extent of coupling between classes residing at the same abstraction layer). So, I have adopted an approach that draws a firm line between process and design. The core component of OO process education is the demonstration of the interrelationships between object models and code. Traceability from object models to code is a key strength of OO and a weakness with Structured Development techniques. This is because classes in a conceptual model map

naturally to classes in OO code, whereas a semantic gap exists when moving from a traditional conceptual model such as a dataflow diagram to Structured Programming code.5 You can best demonstrate this traceability using an OO CASE tool. I supply a simple working application in C++, consisting of four classes and approximately 100 lines of code. I ask developers to create an object model in a CASE tool such that the generated code exactly matches the supplied code (with the exclusion of method bodies). I ask developers to strictly avoid discussing issues about the application’s semantics or the design philosophy. OO design education commences only after the developers successfully complete this exercise; this helps because design issues can’t then distract them! OO design education is considerably more difficult. Principles such as the Liskov Substitution Principle tend to be somewhat “heavy,” and progress along the learning curve is further stymied because developers insufficiently appreciate that OO design is a distinct area of knowledge. Developers have the dangerous tendency to want to get on with the design rather than wait till the mentor has proclaimed them competent to do it. To a large extent, this is an instance of “unconscious incompetence” (where a person does not have the requisite skills and does not know it). Using design posers addresses this problem, but only to some extent: developers often see the posers as not life-sized and hence not serious enough to worry about. As I explain later, where possible I defer design education till after finalizing the object model and the key public-method interfaces. This strategy effectively supplies a life-sized design poser to the mentees, thus providing an opportunity to initially learn by watching rather than by doing. They can later apply this learning to the same poser (typically while addressing requirement changes) with only periodic hand-holding. Reflections. Draw a firm line between process education and design education, and ensure, before beginning design education, that mentees internalize the correspondence between models and code. Expect that designers new to OO are in a state of unconscious incompetence. Have a tactical plan to move them to a state of conscious incompetence before beginning design education.

Managing design The object design phase is the most crucial phase in the OO project. Compared to a traditional function-oriented design, an OO design produces a relatively large number of interfaces with a correspondingly small method size. This has two main consequences. First, a mediocre design wreaks much more havoc during and after implementation in an OO context than does a downright bad design in a traditional context. Second, the design-to-code effort ratio is typically much greater in an OO context (in the region of 1:1) than in a traditional context (in the region of 3:7). These factors increase the need for good design management. Typically, you can split the object design phase into two distinct subphases that require different perspectives and skills. The first is architectural or high-level design; the second is detailed or low-level design.2 This division is particularly important for the mentor in a project team that has little OO design experience. The mentor, who is more knowledgeable on design issues, creates and owns the architectural design. The mentees create and own the detailed design, albeit with periodic hand-holding by the mentor. This, as I have already noted, lets mentees learn initially by watching rather than by doing. They can apply their learning to subsequently change the architectural design, if required, in consultation with the mentor. This division also stabilizes the design process because the design’s foundation—the object model and key architectural decisions—is not stirred up repeatedly on the turbulent journey up the learning curve. An interesting and frequent question is, “Should the OO design apprentice be associated with the mentor during architectural design?” I believe not, unless the project is an internal pilot with the freedom to fail. The design activity can be quite iterative and subjective—and thus a matter of personal pride—even for the experienced architect, let alone the (unconsciously incompetent) apprentice. So, discussion of work in progress often requires the mentor to devote much time to explicitly evaluating the apprentice’s designs (which are usually quite naive) and comparing them with his own. Consequently, a learner’s first exposure should be to completed (or nearly so) work products of architectural design, not to work in progress.

Expect that designers new to OO are in a state of unconscious incompetence.

May/June 2001

IEEE SOFTWARE

39

Because the apprentice is not initially familiar with the architectural work products, they can double as life-sized design posers. They form a convenient platform for the mentor to launch design education and to demonstrate the lessons embodied in earlier design posers. Design principles are debated more hotly and internalized better when argued over a real-life poser that comes with the responsibility for customer delivery! Reflections. The mentor must own and perform the architectural design and must avoid discussing work in progress with mentees. Expect the transition to conscious incompetence to be complete only when the architectural design has been understood, accepted, and finalized. Implementation In general, my involvement as an OO mentor is minimal during implementation. It has occurred only when the architectural design or a complex piece of detailed design requires some clarification or change. However, the mentor does need to help the team judiciously apply the philosophy of iterative development. A project manager going by the book would probably be tempted to analyze, design, and code in a highly iterative manner. This advice is not universally applicable; the mentor must assess the degree of iteration required for the occasion. For example, for reengineering and redesign projects with low technology risks and low requirements volatility, I insist on putting enough meat into the object model so that the team can generate good-quality code before moving into implementation. Predictably, teams tend to lose patience with this waterfall-like approach. However, it works very well, provided the project manager appreciates that the team, despite its limited design exposure, is better off handling bugs and design changes during the design stage, when things are usually under control, rather than during implementation, when they aren’t.

About the Author Ramkumar Ramaswamy is a senior research associate and the head of the Software Concept Laboratory at Infosys Technologies Limited, a CMM Level 5 software services and consulting firm headquartered at Bangalore, India. His areas of interest include methodologies for systems analysis and design, software architecture, and software performance engineering. He received his PhD in operations research from the Indian Institute of Management, Calcutta, and his MSc in physics from the University of Delhi. He is a member of the ACM and the IEEE Computer Society. Contact him at the Software Concept Laboratory, Infosys Technologies Ltd., Electronics City, Bangalore 561 229, India; [email protected].

40

IEEE SOFTWARE

May/June 2001

My involvement during testing has been minimal, too. This might sound surprising because testing OO programs can be more difficult and subtle than testing traditional programs. To understand this, you must appreciate that for typical business applications, even if mission critical, developers see testing as largely intuitive. I routinely come across development teams that do acceptable testing with relatively little recourse to formal, published testing methods. Once a team overcomes its initial discomfort with an OO design’s look and feel, it proceeds to develop its own testing procedure and define its units of test and the integration sequence. The mentor steps in only for reviews of the test plan.

T

he mentor’s role is multifaceted: it combines training, in-project participation, and offline hand-holding. At the project level, the mentor must be the architect and design consultant, as well as a process and (to some extent) language educator. As I mentioned before, the mentor’s involvement will likely be particularly high during architectural and detailed design. The involvement during implementation and testing, however, will likely wane, except possibly when a formal OO testing methodology is mandatory. Whether a single mentor should then assume the additional role of testing methodologist deserves further exploration.

Acknowledgments I thank Vivekanand P. Kochikar, Abdul Sakib Mondal, and the anonymous referees for their comments.

References 1. M.J. Schedlbauer, “The Benefits of Mentoring,” J. Object-Oriented Programming, vol. 10, no. 1, Mar./Apr. 1997, pp. 28–29. 2. G. Booch, Object Solutions: Managing the ObjectOriented Project, Addison-Wesley, Reading, Mass., 1996. 3. L. Hohmann, “The First Step in Training: Analysis and Design or Implementation Language?” J. Object-Oriented Programming, vol. 9, no. 6, Oct. 1996, pp. 61–63. 4. E.F. Gehringer and M.L. Manns, “OOA/OOD/OOP: What Programmers and Managers Believe We Should Teach,” J. Object-Oriented Programming, vol. 9, no. 6, Oct. 1996, pp. 52–60. 5. T. Love, Object Lessons: Lessons Learned in ObjectOriented Development Projects, SIGS Books, New York, 1993.

focus

organizational change

What Makes Measuring Software So Hard? Stan Rifkin, Master Systems

e often hear that it is difficult to get software measurement into practice. Traditional measurement addresses the decisions that support increased quality, increased programmer productivity, and reduced costs—key elements for organizations strategically focused on operational excellence. But what if the organization’s highest priority isn’t operational excellence? This article shows that such organizations have different measurement needs and presents ideas on how to

W Implementing software measurement can be difficult if an organization’s market strategy fails to align with traditional measurements. The author points out the need to understand an organization’s strategic objectives and then tailor its measurement practices to fit. 0740-7459/01/$10.00 © 2001 IEEE

address those needs—thereby making measurement more appealing. While the disparity discussed here involves measurement, it applies to all areas of software process improvement. For example, the Software Engineering Institute’s Capability Maturity Model for Software is silent on two of the three strategies of high-performing organizations—customer intimacy and product innovation. Like traditional measurement, the Capability Maturity Model applies only to organizations wanting to be operationally excellent. How top organizations do it Michael Treacy and Fred Wiersema, in The Discipline of Market Leaders, conducted a survey of 80 top organizations to find out how they out-achieved their competitors.1 Their survey revealed that these companies needed to focus on only one of

three market disciplines—operational excellence, customer intimacy, or product innovativeness—to succeed. Operationally excellent organizations have a formula for their service or product. They offer a small and limited menu of choices, but they deliver excellently and at a competitive price. McDonald’s and Federal Express focus on operational excellence. Customer-intimate organizations seek a different market niche—a total solution.2 Whatever the customer wants they add to the menu. These organizations offer a long, custom-made menu for each engagement. Some financial-service institutions might call customer intimacy a way of getting a greater share of the customer’s wallet, offering spending alternatives ranging from investment opportunities to travel services. Product-innovative organizations pride themselves on maximizing the number of May/June 2001

IEEE SOFTWARE

41

We need to see that the potential rejection of our measurement efforts might be an appropriate response to measures that do not fit their strategy.

turns they get in the market. They introduce many new products, selling innovation and features as opposed to, say, price. Intel, 3M, Sony, and Bell Labs focus on product innovation. They measure their success by the number of new product introductions, the number of patents, and the number of Nobel prizes. Treacy and Wiersema point out that a successful organization must have threshold characteristics of all three disciplines, even though it focuses on and excels at only one. They cite IBM as a company that at one time didn’t have the right balance of the three. IBM focused on customer intimacy and failed to pay attention to price (operational excellence). Competitors not as strong in customer intimacy gained inroads to IBM customers through lower prices. Traditional measurement falls short Traditional software measurements— those espoused by the Software Engineering Institute and Quantitative Software Management, for example—apply almost exclusively to organizations focusing on operational excellence. They typically have little to offer customer-intimate and product-innovative firms. The IT or IS departments in many software development organizations strive for customer intimacy and will do virtually anything their clients request. They get to know their clients very well, sometimes better than the clients know themselves. For example, a payroll service that has seen every variation of payroll processing ends up knowing more about the process than an in-house payroll department does. It could take over its customers’ payroll departments. Microsoft’s focus is product innovation. It touts its new, glitzy features, not its up time or reliability. It wants to earn and own its clients based on new features, not by offering operationally excellent software. Measurement and strategy: four cases What we are missing is a more global view, one that listens to and responds to our measurement customers. We need to see that the potential rejection of our measurement efforts might be an appropriate response to measures that do not fit their strategy. We need to problem-solve jointly with our

42

IEEE SOFTWARE

May/June 2001

clients to develop new classes of measures that simultaneously meet our high standards for objectiveness and their high standards for relevance. Now let’s look at four cases where we’ve tried to implement this view. Wall Street brokerage house In one project, I worked with a prominent Wall Street brokerage house to develop new classes of measures. Their mergers and acquisitions people were interested not in software costs or quality, but in time to market. During the frantic time they were putting a deal together (such as an initial public offering), they needed the IT people to respond quickly so that the firm could earn as much as possible by offering as many services as they could. It was a question of wallet share—a customer-intimate approach. We recognized that the traditional measures of schedule and budget variance would be meaningless in such a context. We agreed on a measure of the percentage of the total deal that did not go to the brokerage. The IT department then offered a realistic plan for continually reducing that (missed wallet share) figure. By clearly focusing on the wallet share, we rejuvenated the almost-dead software process improvement program in the mergers and acquisitions department, and we put new life into the corporate measurement program. Civilian government agency In another case, my client—a computeroriented defense contractor—was developing software for a government civilian agency. The agency wanted project measures, but the agency’s projects were not managed in the traditional way and therefore could not be measured traditionally. The government agency wanted the contractor to do what it requested, not study the request and offer alternatives or pushback. The agency was not as focused on cost, quality, or duration as it was on getting what it wanted in reasonable terms. This customerintimate approach made the menu of services as long as the customer requests. Naturally, the contractor had to deliver the systems within a threshold value of cost, quality, and duration, but customer responsiveness was the primary focus. Because the government agency often

changed its mind about requirements, rendering previous work inapplicable, the contractor faced rework that negatively affected cost and schedule compliance. The agency’s audit department noticed that the part of the agency that ordered software and paid for it was not taking the contractor to task for missing deadlines and budget estimates. The contractor had to respond to the agency’s audit department, but knew it couldn’t use traditional project measurements. I recommended several measures: ■

■ ■ ■

of the total spent by the customer, how much went to competitors (to be minimized); time spent in adversarial encounters (to be minimized); time spent with the customer understanding its business (to be maximized); and the number of people on staff with credentials like the client’s (to be maximized).

This set placed traditional project measurement on a second tier, managed by a separate office, and focused day-to-day programming on customer intimacy. The company implemented these measures immediately because of their high face value. Computer services contractor Another case involved a computer services firm that provided a civilian government agency with the computer programming and operations for a particular payment made to qualified applicants. Because the contract was up for renewal, the computer services firm wanted to propose adopting a set of measures that would indicate the firm’s operational excellence. The contractor and the agency failed to come up with measures that resonated with the programming and operations staff, even though the measures came from the Practical Software Measurement Support Center (www.psmsc.com). It turned out that the government agency was really focusing on product innovation. It felt behind the times in terms of technology and really wanted a new, modern IT provider, not a better, cheaper, faster provider of old technology. In fact, there was no business driver for the desire for more modern technology, only a (vague) belief that such technology would reap financial benefits to the government in terms of potentially lower

costs and greater flexibility. The measures we agreed on were ■ ■

■

planned versus actual implementation of a set of new technology introductions; hours spent training the government client on the principles of that new technology; and reliability measures directly related to the government organization’s business, such as the cost of government rework due to provider payment errors, idle government worker hours due to system downtime, and government time spent in meetings or on the phone with qualified applicants due to provider service failures.

Clearly, market disciplines other than operational excellence have different process and measurement needs.

These measures replaced the traditional measures previously used, such as percent of system availability, data entry error rates, and a threshold number of abnormal operational terminations per day. Neither the contractor nor the government agency had heeded the traditional measures, which did not relate to the government mission or daily reality. The computer services firm implemented the new measures as fast as they were communicated. Again, they had high face value. The nonprofit world Creating software measurement that aligns with an organization’s primary value proposition applies to nonprofit organizations as well. For example, the United Way of America has adopted product innovativeness as its primary value proposition because it believes that product innovation is the only sustainable strategy for a national charity. Measuring according to the right strategy Clearly, market disciplines other than operational excellence have different process and measurement needs. Here are a few ideas from my practice about how to address those needs. Customer intimacy Customer-intimate organizations seek flexibility so that they can extend their menus infinitely. To align software measurement with customer intimacy, we must measure flexibility and wallet share. For exMay/June 2001

IEEE SOFTWARE

43

We need to develop a whole new set of measures for all those customerintimate and productinnovative organizations that have avoided measurement thus far.

ample, in peer reviews we need to closely examine the elements that limit future options, such as a limit on the number of items in a list and built-in, hard-coded “magic” numbers. Also, we need to judge comprehension during reviews because the artifacts will constantly be expanded and enlarged as a strategy and must be understandable. Accordingly, we would seek measures of comprehension and understandability as surrogates for maintainability and changeability. We should measure configuration management for customer-intimate organizations by the number of interfaces they manage. After all, we seek a plug-and-play architecture where we can remove and replace components, worrying only that the components obey the interface requirements. Probably the most important ingredient of a customer-intimate system is a systems architecture, so a simple, appropriate measure would be counts of architecture checks and violations. Product innovativeness Product-innovative organizations concentrate on features over quality, reliability, cost, and flexibility (unless those are the specific features being optimized, which is rare). Users of innovative products have a certain patience required with new products, such as the Palm Pilot, Walkman, Watchman, wearable cell phone, Linux, and Windows 2000. Innovative organizations often let traditional planning fall by the wayside. They value innovation as more important than planning; plans are not a deliverable. Their planning often takes the form of creating a diversity of investment alternatives, assuming that some “bets” will fail to pay off. We see this particularly in pharmaceutical firms. Their management doesn’t require researchers to discover a particular drug by a particular deadline, but focuses on regular discoveries in the pipeline and, on balance, a healthy proportion of winners. The challenge here for those of us who care about process is to create lightweight, generic processes that can be applied with large helpings of intelligence and judgment.3 As much as those of us with a process focus might hate to hear it, innovative organizations require only “good enough” quality. Features, not quality, are the deliverable. Therefore, quality goals should focus on

44

IEEE SOFTWARE

May/June 2001

thresholds, benchmarks, and especially time to market. Our measure here should be comparative: how does our quality stack up against those we view as competing for our market share? Companies with product-innovative (or customer-intimate) strategies are organized differently than those with an operationalexcellence strategy. Product-innovative organizations have high differentiation (meaning many experts) and high integration (getting disparate, possibly competing experts to serve in the interests of a common, corporate goal). One measure I use is a count or proportion of the number of people in the organization whose job is to integrate those competing interests to make a product happen. In Microsoft’s applications area (office and programming-language products), such people head 10-person teams, so both the count and ratio are high relative to customer-intimate and operationally excellent firms. GQM to the rescue? Those of us in the measurement game know the Goal/Question/Metric process well and use it regularly.4,5 GQM is a method of cascading from business goals to decisions needing information, to determining what to measure to supply that information. Because GQM flows down from goals, we might assume that it reflects the choice of market disciplines. In fact it does, provided that the GQM practitioner knows the three market disciplines and probes the management about them. Too often the GQM practitioner listens to management and is a faithful scribe but fails to question the focus, balance, and alignment of goals with strategy. I can find no reference to GQM that indicates a thorough examination of organizational strategy; business goals yes, strategy no.

S

o where does this leave us? Measuring operational excellence is more or less a solved problem. We need to develop a whole new set of measures for all those customer-intimate and product-innovative organizations that have avoided measurement thus far. When we do, implementing software measurement will be as easy as implementing strategy. In organiza-

tions where there is a fit between measurement and strategy, we will be able to implement measurement programs without people hating us. Acknowledgments I learned most of this by working with John Title of Computer Sciences Corporation. The measurement leader who made me ask myself many of these questions is David Card, whose measurement leadership I have always appreciated. I gratefully acknowledge audiences at the 1999 NASA SEL Software Engineering Workshop and 2000 SEI Software Engineering Process Group National Meeting for their feedback, to David Blaine, and especially to IEEE Software’s editor in chief and anonymous reviewers.

References 1. M. Treacy and F. Wiersema, The Discipline of Market Leaders: Choose Your Customers, Narrow Your Focus, Dominate Your Market, Addison-Wesley, Reading, Mass., 1995. 2. F. Wiersema, Customer Intimacy: Pick Your Partners, Shape Your Culture, Win Together, Knowledge Exchange, Santa Monica, Calif., 1996.

3. M. Fowler, “Resources to Slim Down Your Software Process,” Software Development, vol. 8, no. 12, Dec. 2000, p. 34. 4. R. van Solingen and E. Berghout, The Goal/Question/ Metric Method: A Practical Guide for Quality Improvement of Software Development, McGraw-Hill, London, 1999. 5. R. Park, W. Goethert, and W. Florac, Goal-Driven Software Measurement—A Guidebook, tech. report CMU/SEI96-HB-002, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 1996; www.sei.cmu.edu/publications/ documents/96.reports/96.hb.002.html (current 12 Apr. 2001).

About the Author Stan Rifkin is a principal with Master Systems, an advisory services firm that specializes in helping organizations for whom computing is strategic. He worked at the Software Engineering Institute on implementing software process improvement and is co-chair of the 2002 Software Engineering Process Group Conference. He was the chief information officer at the American Association for the Advancement of Science and the head of systems development at the National Headquarters of the American Red Cross. He has a BS in business administration, an MS in computer science, and is completing a doctorate in education. He is a member of the IEEE Computer Society, the ACM, the Foundation for the Empirical Study of Programmers, the Academy of Management, the Institute for Operations Research and Management Science, the Project Management Institute, and the Future Search Network. He is on the editorial board of Empirical Software Engineering. Contact him at [email protected].

2001

Call for Papers IEEE Intelligent Systems seeks papers on all aspects of artificial intelligence, focusing on the development of the latest research into practical, fielded applications. Papers should range from 3,000 to 7,500 words, including figures, which each count as 250 words. Submit one double-spaced copy and a cover letter or e-mail to Magazine Assistant IEEE Intelligent Systems 10662 Los Vaqueros Circle PO Box 3014 Los Alamitos, CA 90720-1314 phone +1 714 821 8380; fax +1 714 821 4010 [email protected] For author guidelines, see http://computer.org/intelligent/author.htm IEEE

& their applications

May/June 2001

IEEE SOFTWARE

45

focus

organizational change

Developing Groupware for Requirements Negotiation: Lessons Learned Barry Boehm, University of Southern California Paul Grünbacher, Johannes Kepler University Linz Robert O. Briggs, GroupSystems.com

efining requirements is a complex and difficult process, and defects in the process often lead to costly project failures.1 There is no complete and well-defined set of requirements waiting to be discovered in system development. Different stakeholders—users, customers, managers, domain experts, and developers—come to the project

D The authors discuss the lessons they learned while developing four generations of a distributed groupware system called WinWin. 46

IEEE SOFTWARE

with diverse expectations and interests. Requirements emerge in a highly collaborative, interactive, and interdisciplinary negotiation process that involves heterogeneous stakeholders. At the University of Southern California’s Center for Software Engineering, we have developed a series of groupware implementations for the WinWin requirements negotiation approach (see the Acknowledgments at the end of the article for a list of organizations that helped sponsor this research). The WinWin approach involves having a system’s success-critical stakeholders participate in a negotiation process so they can converge on a mutually satisfactory or win–win set of requirements. Groupware-supported methodologies are among the hardest to get right, and the rapidly moving technology of distributed inter-

May/June 2001

active systems is a major challenge. This is due largely to the relative newness of the area and to the unfamiliarity of most software developers with the phenomena of group dynamics. However, an even bigger challenge is creating a system that works well with people of different backgrounds, in different places, and often at different times. In particular, collaborative technology that supports requirements negotiation must address stakeholder heterogeneity. Our WinWin groupware system—which has evolved over four generations—enables and facilitates heterogeneous stakeholder participation and collaboration. Each generation reflects an increase in our understanding of what is needed for successful WinWin groupware operations and technology support. Here, we present the major lessons we learned during WinWin’s development. 0740-7459/01/$10.00 © 2001 IEEE

The WinWin approach The original motivation for a WinWin groupware system was Barry Boehm’s frustration in using a manual win–win approach to manage large projects at DARPA. For example, win–win management of the $100million DARPA STARS program was done primarily through monthly meetings of many STARS stakeholders: three prime contractors and their three commercial counterparts; three user representatives from the Army, Navy, and Air Force; DARPA customers and contract managers; and several research and support contractors. Each meeting concluded with a win–win agreement, so after a meeting, participants felt they had taken three steps forward. However, by the next meeting, the distributed stakeholders had independently “reinterpreted” the agreements, causing the process to move two steps back. As a result, it took six months to achieve a shared vision that the prime contractors’ success plans documented. Our analysis at the time indicated that a WinWin groupware support system could reduce this process to one or two months. The general win–win approach evolved more or less independently as an interpersonal-relations,2 success-management,3 and project-management4 approach. We usually define it as “a set of principles, practices, and tools, which enable a set of interdependent stakeholders to work out a mutually satisfactory (win–win) set of shared commitments.” Interdependent stakeholders can be people or organizations. Their shared commitments can relate to information system requirements in particular (the WinWin groupware system’s primary focus) or can cover most continuing relationships in work and life (for example, international diplomacy). Mutually satisfactory generally means that people do not get everything they want but can be reasonably assured of getting whatever it was to which they agreed. Shared commitments are

not just good intentions but carefully defined conditions. If someone has a conditional commitment, he or she must make it explicit to ensure all stakeholders understand the condition as part of the agreement. Why does WinWin work? WinWin works because people and groups have different preference patterns. A classic example was the 1978 EgyptianIsraeli peace treaty’s negotiation of the Sinai Peninsula borderline. It was at an impasse until it was clarified that Egypt preferred territory and Israel preferred getting a demilitarized zone. We elaborate on other reasons why WinWin works in the following.

In particular, collaborative technology that supports requirements negotiation must address stakeholder heterogeneity.

Win–lose doesn’t work In requirements negotiation, nobody wants a lose–lose outcome. Win–lose might sound attractive to the party most likely to win, but it usually turns into a lose–lose situation. Table 1 shows three classic win–lose patterns among the three primary system stakeholders—developers, customers, and users—in which the loser’s outcome usually turns the two “winners” into losers.5 As the table shows, building a quick and sloppy product might be a low-cost, near-term win for the software developer and customer, but the user (and maintainer) will lose in the long run. In addition, adding lots of marginally useful bells and whistles to a software product on a cost-plus contract might be a win for the developer and users, but it is a loss for the customer. Finally, “best and final offer” bidding wars that customers and users impose on competing developers generally lead to lowball winning bids, which place the selected developer in a losing position. However, nobody really wins in these situations. Quick and sloppy products destroy a developer’s reputation and have to be redone—inevitably at a higher cost to the customer. The bells and whistles either disappear

Table 1 Frequent Software Development Win–Lose Patterns (That Usually Turn into Lose–Lose Situations) Proposed solution

“Winner”

Loser

Quickly build a cheap, sloppy product Add lots of “bells and whistles” Drive too hard a bargain

Developer and customer Developer and user Customer and user

User Customer Developer

May/June 2001

IEEE SOFTWARE

47

A WinWin approach builds a shared vision among stakeholders and provides the flexibility to adapt to change.

or (worse) crowd out more essential product capabilities as the customer’s budgets are exhausted. Inadequate lowball bids translate into inadequate products, which again incur increased customer costs and user delivery delays to reach adequacy. WinWin builds trust and manages expectations.

If you consistently find other stakeholders asking about your needs and acting to understand and support them, you will end up trusting them more. In addition, if you consistently find them balancing your needs with other stakeholders’ needs, you will have more realistic expectations about getting everything you want. WinWin helps stakeholders adapt to changes.

Our traditional, adversarial, lawyer-oriented contracting mechanisms are no match for our current world of increasingly rapid change in technology, mergers, reorganizations, and personnel turnover. Instead of rigorous requirements in ironbound contracts, doing business in Internet time requires stakeholders with a shared vision and the flexibility to quickly renegotiate a new solution once unforeseen problems or opportunities arise.6–7 A WinWin approach builds a shared vision among stakeholders and provides the flexibility to adapt to change. WinWin helps build institutional memory. The why behind the what—that is, the decisions that lead to a work result—often vanish. By capturing and preserving stakeholder negotiations, WinWin supports long-term availability of the decision rationale and thus helps build institutional memory. Having more auditable decisions creates more detailed, accurate, and complete deliverables.

How does the WinWin negotiation model work? The particular approach we have evolved includes a WinWin negotiation model for converging to a win–win agreement and a WinWin equilibrium condition to test whether the negotiation process has converged. The negotiation model guides success-critical stakeholders in elaborating mutually satisfactory agreements. Stakeholders express their goals as win conditions. If everyone concurs, the win conditions become agreements. When stakeholders do not concur, they identify their conflicted win conditions 48

IEEE SOFTWARE

May/June 2001

and register their conflicts as issues. In this case, stakeholders invent options for mutual gain and explore the option trade-offs. Options are iterated and turned into agreements when all stakeholders concur. Additionally, we use a domain taxonomy to organize WinWin artifacts, and a glossary captures the domain’s important terms. The stakeholders are in a WinWin equilibrium condition when the agreements cover all of the win conditions and there are no outstanding issues. Four generations of tool support The WinWin negotiation model provided the basis of all four generations of WinWin groupware systems. G1: Initial prototype The first-generation WinWin groupware implementation was a prototype developed in concert with Perceptronics’ CACE-PM support system for concurrent engineering of multichip modules. CASE-PM let us develop a useful prototype in several weeks, which was sufficient for demonstrations and an initial experiment. This involved having the G1 WinWin system developers perform role-playing as future system developers, customers, and users negotiating the requirements for a more robust version of WinWin. Performing the WinWin negotiation with G1 WinWin gave us a strong, shared vision for the system’s next version, validating its utility as a groupware capability. G2: Strong vision, not-so-strong architecture The second-generation WinWin system used a Sun-Unix client-server architecture, X/Motif GUI support, and its own database server. Some friendly industry users tried it experimentally. G2 WinWin’s main value was identifying inconsistencies between the negotiation model and the artifacts, among the artifacts, and between the GUI and the database server. However, we underestimated how much detailed software engineering it would need to get from a shared groupware vision to a groupware support system. G3: Muscle-bound architecture The third-generation WinWin system had a formally analyzed negotiation model, a uniform artifact look and feel, carefully defined GUI-database interfaces, and rigorous enforcement of the negotiation model.

What Is a Group Support System? Some group tasks can be most effectively accomplished by carefully coordinated individual efforts. Technologies that support this kind of teamwork abound—email, team calendaring, shared document repositories, and so on. Other tasks, such as requirements negotiation, require concerted reasoning by many minds. For such tasks there are Group Support Systems. On the surface, a GSS might seem like a collection of glorified chat tools with some voting tools thrown in for good measure. For example, most GSS suites include shared list tools. Any user can make a contribution to a shared list at any time, and any contribution a person makes appears instantly on all the other users’ screens. Various GSS suites include shared outlines, shared comment windows, shared drawing tools, and so on. In each tool, all the users can talk at once, contributing to the discussion as inspiration strikes rather than waiting for the floor. GSS suites usually include a variety of useful voting tools, including Likert scales, semantic anchors, allocation votes, and multicriteria votes. The users can move their contributions into a vote tool, evaluate them, and then instantly review their results online. The real magic of a GSS is not what you can make happen on the screen but what you can make happen in the group. Using a GSS, you can create predictable, repeatable patterns of human interaction and reasoning among people working toward a common goal. For example, most GSSs include brainstorming tools that can help a group diverge from customary patterns of thinking. Some GSSs also have idea-organizing tools that let a group structure disorganized ideas. Other GSS tools can help a group converge quickly from its many brainstorming ideas down to a clear focus of just the ideas that merit further attention. Using a GSS, you can arrange a sequence of steps that a team can follow as they reason together to accomplish their task. In each step, the GSS tools are configured so that as participants contribute to the system, a useful pattern of thinking emerges. In all, there are seven basic patterns of thinking a GSS can create in a group:1 Diverge: Move from having fewer ideas to more ideas, Converge: Move from having many ideas to focusing on just the few that are worthy of further attention, ■ Organize: Start to gain a better understanding of the relationships among ideas, ■ Elaborate: Start expressing ideas in more detail, ■ ■

It also had a number of amenities for voting, for attaching associated documents or analysis-tool runs, and for big-picture negotiation visualization and navigation. Its major problems were its insufficient robustness and the overly strict enforcement of the negotiation approach that kept it from adapting to different negotiation situations. G4: Group support system infrastructure Our experiences with the first three generations of WinWin encouraged USC to develop a version of WinWin based on

Abstract: Move from expressing ideas in detail to expressing them as fewer, more general concepts, ■ Evaluate: Start better understanding the value of concepts for accomplishing the task at hand, and ■ Build consensus: Start better understanding the diverse interests of the group members and begin to agree on possible courses of action. ■

Besides requirements negotiation, GSS processes have been implemented for a variety of organizational tasks that require many people to think together. Here are a few examples: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

strategic planning, new product development, marketing focus groups, total quality management, military intelligence analysis, organizational change management, data modeling, group therapy, factory floor design, and software inspections.2

Because a GSS operates over a computer network, team members can often interact, even when oceans and continents separate them. However, just as a screwdriver is not very useful for pounding nails, a GSS is not right for every group interaction. Sometimes a team still needs to get together the old-fashioned way, eye-to-eye, to see who sweats and who blinks first. Nonetheless, extensive research shows that under the right circumstances, teams using a GSS can reduce their labor hours by 50 percent or more and cut their project cycles by 60 to 90 percent. Such teams usually also report a higher-quality result than they were able to obtain using more conventional means.3

References 1. R.O. Briggs, G.-J. de Vreede, and J.F. Nunamaker, Jr., “Thinklets: Achieving Predictable Repeatable Patterns of Group Interaction with Group Support Systems (GSS),” Proc. HICSS 2001 (Hawaii Int’l Conf. System Sciences), IEEE CS Press, Los Alamitos, Calif., 2001. 2. M. van Genuchten et al., “Industrial Experence in Using Group Support Systems for Software Inspections,” IEEE Software, vol. 18, no. 3, May/June 2001, pp. 60–65. 3. J. Fjermestad and R. Hiltz, “Case and Field Studies of Group Support Systems: An Empirical Assessment,” J. Management Information Systems, to be published, 2001.

the commercial groupware infrastructure GroupSystems.com developed in cooperation with the University of Arizona.8 Our current collaboration between USC and GroupSystems.com has led to a fourth-generation system, called EasyWinWin.9 EasyWinWin EasyWinWin is a requirements definition approach based on a Group Support System. A GSS is a suite of software tools that can create, sustain, and change patterns of group interaction in repeatable, predictable ways May/June 2001

IEEE SOFTWARE

49

Figure 1. An example of how a team builds a clean list of win conditions and organizes them into predefined buckets.

(see the “What is a Group Support System?” sidebar for details). The tools help increase stakeholder involvement and interaction. EasyWinWin defines a set of activities guiding stakeholders through a process of gathering, elaborating, prioritizing, and negotiating requirements. In addition, it uses group facilitation techniques that collaborative tools support. Table 2 summarizes the main stakeholder negotiation activities involved in using EasyWinWin, and how they are implemented through group techniques and thinking patterns that the GroupSystems.com GSS sup-

Figure 2. In EasyWinWin, red cells indicate a lack of consensus. The cell graph shows a voting pattern used to trigger an oral discussion revealing unshared information, unnoticed assumptions, hidden issues, constraints, and so forth. This particular graph shows the voting pattern for win condition 2.1, with criterion Business Importance. 50

IEEE SOFTWARE

May/June 2001

ports. Figure 1 shows an EasyWinWin example of candidate win conditions for a USC bookstore Web portal requirements negotiation and how stakeholders can categorize them by dragging them into buckets. In contrast, 3G WinWin had a much more formal way of defining win conditions, in which categorization involved scrolling through a category list in another window and typing in the category name or number. With 3G WinWin, it would usually take about a day to enter and categorize about 15 to 25 win conditions. With EasyWinWin, it would take about two hours to enter and categorize

Table 2 EasyWinWin Activities, Group Techniques, and Patterns of Thinking (see the sidebar for information on patterns of thinking) Activity

Purpose

Group technique

Review and expand negotiation topics

Stakeholders jointly refine and customize an outline of negotiation topics based on a taxonomy of software requirements. The shared outline helps stimulate thinking, organize win conditions, and check negotiations.

Stakeholders add comments and Diverge recommend changes to the outline. ____________________________________________

Brainstorm stakeholder interests

Converge on win conditions

Stakeholders share their goals, perspectives, views, background, and expectations by gathering statements about their win conditions. Stakeholders jointly craft a nonredundant list of clearly stated, unambiguous win conditions by considering all ideas contributed in the brainstorming session.

Capture a glossary of terms

Stakeholders define and share the meaning of important terms of the project/domain in a glossary of terms.

Prioritize win conditions

The team prioritizes the win conditions to define the scope of work and to gain focus.

Reveal issues and constraints

Stakeholders surface and understand issues.

Identify issues, options, agreements

Identify the issues that arise owing to constraints and conflicting win conditions. Propose options to resolve these issues. Negotiate agreements.

about 50 to 120 win conditions. Figure 2 shows EasyWinWin’s capability for group prioritization of win conditions. Stakeholders can quickly see which win conditions are more and less important and easy to implement, plus where their degree of consensus is stronger (the green and red cells). With 3G WinWin, we had an awkward interface to a separate and partially implemented prioritization tool that was rarely used. EasyWinWin has been used in about 30 real-world projects. We applied the approach in various domains (for example, in digital libraries, an e-marketplace, and col-

Thinking pattern

A moderator reviews these comments together with the group and modifies the outline Free-format brainstorming: Anonymous, rapid brainstorming on electronic discussion sheets

Converge

Diverge

Fast focus: A structured Converge discussion to converge on key win conditions ____________________________________________ Categorize win conditions Organize into negotiation topics. Stakeholders propose initial Elaborate definition of terms based on stakeholder statements. The team then jointly reviews and agrees on the terms. Stakeholders rate win conditions Evaluate for each of two criteria: business importance (relevance of a win condition to project/ company success) and ease of realization (perceived technical or economic constraints of implementing a win condition). Crowbar: Analyze prioritization Build consensus poll to reveal conflicts, constraints, different perceptions, and so forth. WinWinTree: Review win conditions, Elaborate identify issues, and porpose options ________________________________________________ Negotiation of agreements

Build consensus

laboration technology) and thoroughly explored and refined the various collaborative techniques with the goal of streamlining the negotiation protocols and the overall order and design of process steps. We captured our experiences in a detailed process guidebook that explains our approach to project managers or facilitators.10 Lessons learned We learned three major types of lessons while developing the four generations of WinWin systems: methodology, groupware, and project. May/June 2001

IEEE SOFTWARE

51

Collaborative technology for requirements engineering must be based on collaboration and facilitation techniques that emphasize group dynamics.

Methodology lessons Groupware tools for information technology requirements negotiation need a methodology that reflects both the evolving role of requirements in the IT life cycle process and the sensitivity involved in successful group dynamics. Here are some examples of methodology lessons we learned. Define a repeatable requirements negotiation process. The first three generations provided

only a top-level strategy on how to carry out a concrete WinWin negotiation. In the EasyWinWin project, we focused on moving people through a process that builds mutual understanding and a shared vision in progressive steps. Each step involves the stakeholders in assimilating each others’ views and in building consensus on a mutually satisfactory shared vision and set of system requirements. A process guide explains the use of group techniques and collaborative tools in a requirements negotiation.10 We found that a detailed process guide reduces variance in the quality of deliverables and helps lowerskilled or less experienced practitioners accomplish more than would be possible with straight stand-up facilitation. Incorporate facilitation and collaboration techniques. Collaborative technology for re-

quirements engineering must be based on collaboration and facilitation techniques that emphasize group dynamics. The first three generations of WinWin environments emphasized modeling constraints over group dynamics and collaboration support. Groupware-supported collaboration techniques adopted in EasyWinWin help create desired patterns of group interaction (see the sidebar). An example is the anonymous submission of stakeholder contributions, such as win conditions or voting ballots used to foster candor. This way, people with power differentials can make proposals without feeling a threat to their job, status, relationships, or political position. Increased openness also helps stakeholders quickly get to the root issues. People up and down the hierarchy are better informed and thus can avoid the Abilene Paradox (in which people agree to an unattractive option because they erroneously believe it will make the option-proposer happy).11 52

IEEE SOFTWARE

May/June 2001

Recognize the role of negotiations in the life cycle. A major lesson learned from the ex-

periment with the 1G WinWin system was that the WinWin approach helps bridge a previous gap5 in using the spiral process model: determining the next round of objectives, alternatives, and constraints. This led to the WinWin spiral model extensions that several organizations now use.12 Experiments also showed that we should perform prototyping ahead of and during requirements negotiations: The 3G WinWin was sufficiently robust to support four years’ worth of projects—with 15 to 20 project negotiations per year.12,13 These project negotiations involved USC librarians and student teams negotiating the requirements for operational USC digital library systems, which the student teams then built and transitioned to library use. In the first year, we learned not to do the WinWin negotiations ahead of the prototype, as we rediscovered the IKIWISI (I’ll know it when I see it) syndrome. Once the librarians saw the prototype, they wanted to redo all the negotiations. In the following years, we verified across over 100 requirements negotiations that 3G WinWin could support rapid definition and development of unprecedented applications. WinWin has been successfully used in various contexts of a requirements definition. This includes the development of a shared vision among stakeholders, requirements definition for custom development projects, COTS acquisition and integration, transition planning, and COTS product enhancement and release planning. Make sure your stakeholder negotiators have the essential characteristics. Your stake-

holder negotiators should be representative, empowered, knowledgeable, collaborative, and committed. We identified these characteristics after analyzing the critical success factors for transition into digital library operational use. Successful win–win negotiations often involve prescreening stakeholder negotiators and performing shared-knowledgebuilding activities such as preliminary teambuilding sessions and concurrent prototyping. Refine agreements into more measurable requirements. The result of a WinWin nego-

tiation is typically not a complete, consis-

tent, traceable, and testable requirements specification. For example, stakeholders might become enthusiastic about proposed capabilities and ratify idealistic agreements such as “anytime, anywhere” service. Rather than dampen their enthusiasm with a precise wordsmithing exercise, it is better to have a facilitator postprocess such agreements into more precise and realistic requirements for the stakeholders to review and iterate. Groupware lessons Some of the strongest agreements among negotiators in our early WinWin systems were about deficiencies in the groupware. This is not where you want your negotiators focusing their attention. Later versions of WinWin became more effective in keeping stakeholders focused on their negotiations. Make use of unobtrusive and flexible tools.

Each of our first three generations of WinWin groupware was increasingly strict about enforcing modeling conventions at the expense of group dynamics. Several industry and government organizations also used 3G WinWin experimentally. However, this use did not lead to the system’s crossing the chasm into mainstream use. The main reason the users cited was that 3G WinWin’s integrity rules were too rigorous. For example, when an agreement was put to a vote, all of its associated win conditions, issues, and options were locked to preserve the voting process’s integrity. Then, to fix a typo in an artifact that was locked for voting, users had to make all the locked artifacts inactive and copy their contents into a new set of artifacts. Furthermore, they could not define issues without win conditions as referents or define options without issues as referents. Thus, the software got in the way of the human interactions, which were a critical part of negotiation. In EasyWinWin, we decided to relax such constraints, with the result that industry and government users have been enthusiastic about their negotiation experiences.

ship management than was needed for multichip modules. This led to a much more thorough definition of WinWin artifacts and relationships, including the basic negotiation model discussed earlier. Focus on ease of use to foster stakeholder involvement. People participating in a require-

ments negotiation typically don’t have time to take training before starting to negotiate. Ease of use lets more people directly participate and elicits more from everybody involved. This leads to better buy-in because more interests can be accommodated earlier in the process. It also helps to develop broader and deeper deliverables. EasyWinWin results surpass 3G WinWin results in terms of the number of artifacts collected in a negotiation.13 In addition, the higher number of issues identified and resolved helps reduce risks early in a project and the chances of it derailing later. For example, our 3G WinWin digital library requirements negotiations involved 15 to 25 win conditions and converged in two to three weeks (as compared to two to three months for comparable manual win–win requirements negotiations). Similar EasyWinWin requirements negotiations involved 50 to 120 win conditions and converged in two to three days. Furthermore, we found that the stakeholders’ experience with EasyWinWin led to better mutual understanding and greater stability of the negotiated requirements.

Using a reliable infrastructure is critical to avoid frustration and to ensure stakeholder buy-in.

Provide a robust infrastructure. The 3G

homemade database server was very fragile and prone to losing people’s work in crashes. Using a reliable infrastructure is critical to avoid frustration and to ensure stakeholder buy-in. Provide interoperability. 3G WinWin did not interoperate well with other groupware systems, even after we built an applications programming interface. One industry project successfully built and applied its own WinWin overlay on top of the groupware system it was using but did not try to develop a more general capability.

Define the negotiation model. Experiments

with the 1G WinWin system showed that software requirements negotiation required considerably more database and relation-

Support for multiple modes of interaction. Be-

yond same-time and same-place stakeholder interaction, we successfully included remote May/June 2001

IEEE SOFTWARE

53

You need to consider how well the user base for which you are building the groupware system represents mainstream end users.

participants in EasyWinWin workshops by using the Web-based capabilities of GroupSystems.com and audio links. We are currently expanding recommendations for geographically distributed teams intending to adopt EasyWinWin activities in different time and place settings. WinWin project lessons Finally, here are some lessons we learned in project organization, user involvement, and expectations management. Involve mainstream end users. With persist-

ence, and by focusing on your mainstream end users, you can develop groupware systems that both speed up the initial definition process and help stakeholders achieve a shared vision with lasting value across the application’s entire life cycle. Thus, you need to consider how well the user base for which you are building the groupware system represents mainstream end users. Once we had an annual set of USC projects to support with 3G WinWin, we overfocused on USC users rather than on our primary target of mainstream industry users. When developing groupware, perseverance pays off. Do not overreact to initial negative

experiences. Groupware systems must be carefully balanced to accommodate the many stakeholders’ different needs. When designing 2G and 3G WinWin, we reacted to the 1G WinWin experience with its highpriced commercial infrastructure by building a homemade infrastructure. The EasyWinWin overlay above GroupSystems.com’s infrastructure has been much more successful. When designing 3G WinWin, we also overreacted to some instances of artifact misuse in 2G WinWin by creating a system whose rules were so rigorous that they turned off most users. 3G WinWin improved on 2G WinWin with its well-defined architectural interfaces but lost out because of its inflexibility for mainstream stakeholder groups. Relative to the “build it twice” guidance in Winston Royce’s initial waterfall model article14 and in Fred Brooks’ The Mythical Man Month,15 you must also add Brooks’ second system syndrome: Developers, particularly for groupware, are likely to react overambitiously to experiences with initial prototypes or systems. 54

IEEE SOFTWARE

May/June 2001

Use the system to plan its own future. Doing this provides both a good test of the current groupware system and a good way of achieving a shared vision of its future directions. Both USC’s experience with using 1G WinWin to negotiate requirements for 2G WinWin and GroupSystems.com’s similar experience with using EasyWinWin substantiates this.

G

SS developers should not expect to get the system right the first time—or even the second time. Our experience with the four generations of WinWin requirements negotiation systems is that it takes several iterations of operational GSSs to fully realize their benefits. Even now, we are involved in a fifth iteration to provide better support for less experienced facilitators. However, the payoffs are worth it: We have experienced about a factor of four improvement in multistakeholder requirements negotiation time when going from manual negotiation to the 2G to 3G WinWin system, and another factor of five in going from 2G3G WinWin to the EasyWinWin system. In addition, the negotiation results have become more thorough and better internalized by the stakeholders.

Acknowledgments DARPA, through Rome Laboratory under contract number F30602-94-C-0195, and the Austrian Science Fund, with an Erwin Schrödinger research grant for Paul Grünbacher (1999/J 1764), sponsored this research. It was also sponsored by the affiliates of the USC Center for Software Engineering: Aerospace, Automobile Club of Southern California, Boeing, CBridge, Chung–Ang University (Korea), Draper Labs, Electronic Data Systems Corporation, Federal Aviation Administration, Fidelity, GDE Systems, GroupSystems.com, Hughes, Institute for Defense Analysis, Litton Industries, Lockheed Martin Corporation, Lucent Technologies, Microsoft, Motorola, Northrop Grumman Corporation, Rational Software Corporation, Raytheon, Science Applications International Corporation, Software Engineering Institute (Carnegie-Mellon University), Software Productivity Consortium, Sun Microsystems, Telcordia Technologies, TRW, US Air Force Research Laboratory, US Army Research Laboratory, US Army TACOM, and Xerox Corporation. We also thank the definers and developers of the first three versions of WinWin: Ellis Horowitz, Dan Port, Prasanta Bose, Yimin Bao, Anne Curran, Alex Egyed, Hoh In, Joo Lee, June Lee, Mingjune Lee, and Jungwon Park. We also thank the users of the four WinWin systems: Frank Beltz, Garry Brannum, Walter

Green, Elizabeth Kean, Judy Kerner, Julie Kwan, Andrew Landisman, Anne Lynch, Ray Madachy, Azad Madni, Nikunj Mehta, Steve Mosher, Karen Owens, Arnold Pittler, Michael Saboe, and John Salasin.

14. W.W. Royce, “Managing the Development of Large Software Systems,” Proc. IEEE WESCON, IEEE Press, Piscataway, N.J., 1970, pp. 1–9. 15. F.P. Brooks, The Mythical Man Month, Addison-Wesley, Reading, Mass, 1975.

References 1. The Standish Group, CHAOS Report, 1995, www. standishgroup.com/visitor/chaos.htm (current 16 Apr. 2001). 2. D. Waitley, The Double Win, Berkeley Books, New York, 1985. 3. S. Covey, The Seven Habits of Highly Effective People, Fireside Books, New York, 1990. 4. B. Boehm and R. Ross, “Theory W Software Project Management: Principles and Examples,” IEEE Trans. Software Eng., July 1989, pp. 902–916. 5. B. Boehm et al., “Software Requirements as Negotiated Win Conditions,” Proc. Int’l Conf. Requirements Eng., IEEE Press, Piscataway, N.J., 1994. 6. B. Boehm, “Requirements That Handle IKIWISI, COTS, and Rapid Change,” Computer, July 2000, pp. 99–102. 7. J. Highsmith, Adaptive Software Development, Dorset House, New York, 2000. 8. J. Nunamaker et al., “Lessons from a Dozen Years of Group Support Systems Research: A Discussion of Lab and Field Findings,” J. Management Information Systems, vol. 13, no. 3, Winter 1996–1997, pp. 163–207. 9. P. Gruenbacher, “Collaborative Requirements Negotiation with EasyWinWin,” Second Int’l Workshop Requirements Eng. Process, IEEE CS Press, Los Alamitos, Calif., 2000. 10. The EasyWinWin Process Guide: USC-CSE and GroupSystems.com, http://sunset.usc.edu/research/WINWIN, 2000 (current 16 Apr. 2001). 11. J.B. Harvey, The Abilene Paradox and Other Meditations on Management, Jossey-Bass, San Francisco, 1988. 12. B. Boehm et al., “Using the WinWin Spiral Model: A Case Study,” Computer, 1998, pp. 33–44. 13. A.F. Egyed and B. Boehm, “Comparing Software System Requirements Negotiation Patterns, J. Systems Eng., vol. 2, no. 1, 1999.

SET INDUSTRY STANDARDS

About the Authors Barry Boehm is TRW Professor of Software Engineering and director of the Center for

Software Engineering at the University of Southern California. His current research focuses on integrating a software system’s process models, product models, property models, and success models via an approach called MBASE (Model-Based Architecting and Software Engineering). He received his BA from Harvard and his MS and PhD from UCLA, all in mathematics. He received an honorary ScD in computer science from the University of Massachusetts. He is an AIAA fellow, an IEEE fellow, an INCOSE fellow, an ACM fellow, and a member of the National Academy of Engineering. Contact him at the USC Center for Software Engineering, Los Angeles, CA 90089-0781; [email protected]. Paul Grünbacher is an assistant professor of systems engineering and automation at the Johannes Kepler University Linz, Austria. His research interests include the application of collaborative technology in software engineering with a focus on requirements engineering and methodologies for software process improvement. He studied Business Informatics and holds a PhD from the University of Linz. Contact him at Systems Engineering and Automation, Johannes Kepler University Linz, Altenbergerstr. 69, 4040 Linz, Austria; [email protected].

Robert O. Briggs is director of methodology and process tools for GroupSystems.com,

where he oversees the future evolution of GroupSystems software, and is also research coordinator at the Center for the Management of Information at the University of Arizona. He investigates the cognitive foundations of collaboration with a focus on the development and deployment of software and processes to enhance the performance of teams making a joint effort toward a goal. He earned his BS in Information Systems and Art History and an MBA from San Diego State University. He earned his PhD in management and information systems from the University of Arizona. Contact him at 1430 E. Fort Lowell Rd. #301, Tucson, AZ 85719; [email protected].

Posix

gigabit Ethernet

enhanced parallel ports

wireless token rings networks FireWire

Computer Society members work together to define standards like IEEE 1003, 1394, 802, 1284, and many more.

HELP SHAPE FUTURE TECHNOLOGIES • JOIN A COMPUTER SOCIETY STANDARDS WORKING GROUP AT

computer.org/standards/ May/June 2001

IEEE SOFTWARE

55

focus

organizational change

Two Interviews, Two Perspectives Ann Miller, Guest Editor

wish I could tell you precisely when and where I first met Eric Schmidt, but I can’t. I can, however, recall two things about our first conversation that are more telling. First, I still have his business card, which reads Vice President and General Manager, Software Products Division, Sun Microsystems. (This tells you something positive about Eric’s previous positions and something less than positive about my business card files.) Second, and more importantly, that first conversation centered on the future of computing, the importance of networking, and the convergence of computing and communications. For this interview, I am reversing my perspective, asking him to reflect back on some of the significant changes at Novell. Whether your company is large, small, or “just right” and whether you are CEO, entry level, or somewhere in the middle, there are insights to be gained from Eric’s thoughtful comments. —Ann Miller

I

Eric Schmidt CEO, Novell 56

IEEE SOFTWARE

Ann Miller: What are some of the stages of organizational change that Novell has gone through, and what were the keys to success at each stage? Eric Schmidt: When I arrived at Novell in April 1997, the company began what I described at the time as a “hardcore turnaround.” We needed to do three basic things: realign Novell’s business with the realities of the networking software marketplace, ship products that we’d promised to customers, and attract top technical talent. May/June 2001

By October 1998, after Netware 5.0 hit the market, we began Phase 2 of the Novell transformation. This involved decoupling our industry-leading directory service from the core product, Netware, and offering it on other platforms (such as NT, Solaris, and, more recently, Linux). We also needed to begin developing and deploying services and solutions that would take advantage of the directory. Moreover, we needed to embark on comprehensive marketing and sales programs that would get 0740-7459/01/$10.00 © 2001 IEEE

these new products and services into the marketplace. Although the period that followed brought great success to the company, we now find ourselves in a slightly different situation and in a different phase in our growth as a leader in network services software. Today, the challenge for the company—from an organizational standpoint—is to increase revenue from the new network services software solutions (such as ZenWorks and BorderManager). We also need to manage the decline in revenue from our traditional core products (such as NetWare). What cultural changes did you lead as Novell grew? The corporate culture that existed at Novell prior to my arrival was one that was eager for leadership—particularly on the engineering side. Having been trained as a computer scientist and having managed many engineering teams at Sun, this was something I knew I could improve. Galvanizing the engineering communities in San Jose as well as in Utah required making several changes to the culture. We needed programs and processes that would give engineers—regardless of where they were in the organization—the freedom to communicate with whomever they wanted to in the company. Although this initially threatened the traditional hierarchy and organizational structure, it had the benefit of surfacing a number of “hidden treasures,” as I like to call them. A few of those projects actually became revenue-generating products.

Again, it took time for the organization to accept this new communications model, but once we started to enjoy the benefits of the new system, there was no looking back. What has been the role of IT software in Novell as the company has evolved? Just as Novell’s product portfolio needed to undergo significant changes during the summer and fall of 1997 (such as moving from the proprietary IPX protocol to open IP), so too did our internal systems. Once we’d made the commitment to move our products to open Internet standards, we began the hard work of moving our payroll, accounting, and sales systems to the Internet as well. Our own technology (namely NDS, now called eDirectory) has helped in a number of ways with this effort. For instance, when a new employee joins Novell, his or her profile is introduced to the system once and populates all the systems and applications that the new employee will use, whether that be the PeopleSoft system, the Oracle systems, or the Siebel sales force automation system. Although we still have much to do, we’re making progress internally to become a showcase for network services software based on directory technology. Eric Schmidt is the chairman of the board and chief executive officer

of Novell. He is actively involved in all significant and strategic decisions and plays a central role in the company’s technical development and management. Prior to heading Novell, he worked at Sun Microsystems (as chief technology officer and corporate executive officer) and as a member of the research staff at the Computer Science Lab at Xerox Palo Alto Research Center. He has an MS in electrical engineering and a PhD in computer science from the University of California at Berkeley. Contact him at www.novell.com.

orporate coach”: Is this just the new millennium term for a corporate consultant? Is this a mentor for CEOs? A psychologist whose client is an entire corporation? I wasn’t certain, so I did the only logical thing: I asked one, and she most graciously replied. Mary Boone has been coaching for some time, and she devotes much of her energy assisting companies that are undergoing organizational transitions. She describes some of the functions of a coach and stresses the interactive nature of communication within a company. You are invited to learn more about corporate coaching, and you won’t even need to ask the coach if you can get in the game. —Ann Miller

C

“

Mary E. Boone President, Boone Associates

May/June 2001

IEEE SOFTWARE

57

Organizational change starts at the individual level. This is why coaching is so important.

58

IEEE SOFTWARE

Ann Miller: Coaching seems to be an emerging trend—but just what is a coach? Specifically, how does he or she differ from the well-established role of management consultant? How does a coach assist in organizational change? Organizational change is of particular interest to me. A sophisticated, multidirectional approach to communication is an absolutely essential element of any organizational change effort. So often in change efforts, we believe that simply keeping people informed is enough. It’s not. Communication in change efforts must be interactive. It’s not enough to simply get “buy in” through the typical corporate change cheerleading effort. We have to get people fully involved so that they own the change instead of simply complying with it. Organizational change starts at the individual level. This is why coaching is so important. To instigate truly interactive communication and change efforts, we must have an interactive mindset. And that means that we have to really involve people in the process of effecting change. This often requires managers and executives to revisit the way they think about things like power and control in management. This isn’t easy for everyone, and that’s why coaching can be such a powerful tool in change management efforts. For example, imagine that an executive in your organization is designated as the head of a knowledge management effort. Her job is to get people to share knowledge throughout the organization. What if she’s the kind of person who plays things extremely close to the vest? How effective can she really be in that role without some coaching? You can have the best interactive technologies in the world, but it won’t make any difference if management lacks an interactive attitude. There are many different types of coaching and many different methods and approaches to the coaching process. My coaching is targeted in a specific arena: communication. This involves much more than how to do a presentation or how to improve your interpersonal skills. I coach people in how to connect, inform, and engage people in organizations by using interactive technologies and interactive methods.

May/June 2001

My coaching is closely connected to my consulting. In my coaching, I usually help an executive see the need for a more interactive management style and give them ideas on how to achieve that. In my consulting, I work with a variety of people in the organization to create a culture where people are connected, well-informed, and fully engaged in their work—whether they’re involved in change management or not. Obviously it’s most effective to be in a situation where you are both coaching the top leadership team and simultaneously working with others in the organization in a consultative relationship. Our readership is international. Are there any significant differences you have seen by country or region, or are we all in pretty much the same situation? Most of my work has been in the US, Canada, and Europe. Certainly there are stylistic differences associated with the cultures of different countries and, of course, different companies. However, regardless of where you are located in the world, the business environment is becoming increasingly complex. This means that the people running an organization (or even part of an organization) cannot be expected to have all the answers regarding strategy and change. Therefore, although it might be somewhat easier to gain people’s compliance with a change effort in a country where the culture is more oriented toward adherence to authority and bureaucracy, it is also true that smart leaders everywhere will recognize that the more input they have to a decision or direction, the more likely they are to succeed. Much has been written about Internet time—information technology companies in the throes of getting Web-based products to market quickly. Are there any special coaching techniques you can recommend for our readers who are in this market? If I were coaching you about getting a product to market quickly, the first thing I would ask you is how you plan to get the customer (or potential customer) involved in that process. There are innovative new ways to apply technology to bring customers into the process of designing a product. And I’m not just talking about a focus group or a survey; I’m talking about real involvement.

Have you considered creating an online team room with a select group of customers? Have you thought of creating a meeting to get key stakeholders from both inside and outside the organization involved? All of us from the IT world know from past disasters with requirements gathering how important it is to have real involvement from key people at a project’s outset. What advice do you have for our practitioner readers who are in the midst of organizational change and who might be firstor second-level managers and aren’t able to call you in for onsite coaching? Well, it’s hard for me to say this without sounding self-serving, but the first thing I’d advise is to buy my book. It’s filled with examples from a variety of companies such as Microsoft, IBM, General Electric, Johnson & Johnson, and many others. It outlines practical actions practioners can take to im-

prove their ability to lead, manage, and cope with change. The other suggestion I have is to consider alternative approaches to coaching. I’ve been experimenting with virtual coaching through phone and email, and it has worked quite well for my clients and me. I have other colleagues who do virtual coaching, and they have found it highly effective as well. Perhaps the most important core message is to consciously attend to communication during change efforts. Whether you are the CEO or a line supervisor, your ability to communicate well will directly affect results.

Whether you are the CEO or a line supervisor, your ability to communicate well will directly affect results.

Mary E. Boone is the president of Boone Associates. She helps leaders

achieve enhanced organizational results through her consulting, speeches, and training in the areas of organizational communication, organizational change, and collaborative technologies. She is also the author of Managing Inter@ctively (McGraw-Hill, 2001). Contact her at [email protected]; www.maryboone.com.

CALL

FOR

IEEE

Articles and Reviewers

Software Security: Building Systems Securely from the Ground Up

Submission deadline: 31 July 2001

Publication: January/February 2002

Fragile and insecure software conti nues to threaten a society increasing ly reliant on complex software syste because most security breaches are ms, made possible by software flaws. Engin eering secure and robust software syste can break the penetrate-and-patch ms cycle of software releases all too comm on today. Topics of interest for this special issue include: • Case studies that help quantify comm on security risks • Security implications of programm ing languages and development tools • Techniques for balancing security with other design goals • Extracting security requirements from software projects • Design for security • Aspect-oriented programming for security • Analyzing programs for vulnerabili ties

• Testing for vulnerabilities • Secure configuration and maintenan ce • Developing trusted environments for running untrusted mobile code • Secure mobile code programming paradigms • Analyzing unknown software for malic ious logic • Intrusion-tolerant software architectu res • Application-based intrusion detec tion • Quantifying trade-offs in adding secur ity during development

Articles must not exceed 5,400 word s including figures and tables, whic h count for 200 words each. Submissio within the theme’s scope will be peer ns -reviewed and edited. Be sure to inclu de the name of the theme for which submitting. Please contact a guest you are editor for more information about the focus or to discuss a potential subm contact the magazine assistant at softw ission; please [email protected] for author guide lines and submission details.

Guest Editors: Anup K. Ghosh, [email protected]; Chuck Howell, [email protected]; and James Whittaker, [email protected]

May/June 2001

IEEE SOFTWARE

59

feature groupware

Using Group Support Systems for Software Inspections Michiel van Genuchten, GroupSupport.com Cor van Dijk and Henk Scholten, Baan Doug Vogel, City University of Hong Kong

oftware engineering sometimes appears to be years behind other disciplines in terms of predictability and quality. But we like to argue that the software industry is facing problems now that other industries have yet to face. Constructing a high-quality, million-line program is daunting. Fortunately, progress is occurring and merging in two important areas: software process improvement and technological support.

S Inspections supported by a group support system can help software professionals detect software defects effectively and efficiently. 60

IEEE SOFTWARE

One example of software process improvement is the development and execution of inspections of software development documents as a mature technique to detect and prevent software defects. Part of an inspection is a meeting where the participants log detected defects and search for more defects. Unfortunately, the effectiveness and efficiency of the logging meeting are often low compared to those of the preparation for the meeting. This has led software engineers and researchers to debate whether an inspection even needs a meeting.1 We approach this question from another angle: can we improve logging meetings such that they contribute to more effective and efficient inspections? Improving meetings through technology has been the main goal of the researchers and practitioners working on electronic meeting systems or group support systems over the last 20 years.2,3 Companies have successfully used these tools for a wide variety of problem-

May/June 2001

solving and business process activities—for example, Doug Vogel has described IBM’s use of an electronic meeting system.4 For more on addressing requirements negotiation, see “Developing Groupware for Requirements Negotiation: Lessons Learned” by Barry Boehm, Paul Grünbacher, and Robert O. Briggs, in this issue. This article describes our experience implementing a GSS for inspections in an industrial environment. The results confirm our belief that such support can improve the efficiency and effectiveness of inspections, provided the inspections are properly conducted. Why GSS support for inspections? An inspection uses standards and checklists to find and fix as many product and process deficiencies as possible. Key characteristics of inspections are individual preparation, data collection, and a fixed syntax to report defects. For those unfamiliar with 0740-7459/01/$10.00 © 2001 IEEE

Inspections

inspections or GSSs, the two sidebars briefly introduce both subjects. During a traditional logging meeting, the participants review the document and state the defects aloud. The reported defects function as triggers to detect more defects. The moderator leads the meeting, and the scribe records all the defects. Experienced groups claim to find many defects during a meeting. You could say that this is a primitive way to consolidate defects. The logging meeting’s two goals (reporting errors and finding new ones) often conflict. The moderator is usually busy controlling the discussion and the logging rate. Inspectors should take the initiative to find new defects, but they are often distracted by other reported defects. This situation suggests that introducing technology into logging meetings might overcome some of these problems. There have, indeed, been several attempts to support software inspections with automated tools. Ilkka Tervonen describes a number of experiments with students, mainly using university prototypes.5 We were also involved in some experiments with GSS-supported inspections with students.6 We concluded that measuring the impact of GSS support is difficult when the group of inspectors is neither trained as inspectors nor experienced in working as a group. As a prelude to operational use, we did two pilot studies at Philips Electronics and Baan in 1996, involving 14 inspections.7 The results in both companies showed that inspectors found more defects and that the logging meeting contributed more to the inspection’s overall result. Furthermore, the engineers appreciated GSS support in the logging meeting. The results triggered the large-scale application of GSS technology at Baan, which we now describe. Applying a GSS at Baan Baan Development is a medium-size industrial software organization developing enterprise resource planning software. GSSsupported inspections started in Baan Development’s Applications Department in the Netherlands, which had 200 engineers. The department had been executing inspections on software documents since 1993. Inspections typically involved three to four engineers, one of whom acted as moderator.

Inspections are a structured way to review software development documents (specifications, the design, or code) with a group of approximately four engineers. Engineers around the world have carried out inspections for more than 20 years. Inspections are widely acknowledged as an important technique to improve software products and processes. An inspection consists of individual preparation by the participants, followed by a meeting in which they log defects and look for more defects, and then reworking of the document by its author. An inspection aims to detect software defects early during development. Inspection results are typically measured in terms of the major and minor defects (majors and minors) that the inspection detected. Majors would, if undetected, result in a defect in test or in the field. Minors are all other defects. Inspection results typically involve three performance indicators. Effectiveness is the number of defects detected per page.1 Efficiency is the number of defects detected per person-hour invested. Effectiveness and efficiency can be calculated for both the preparation and meeting. Yield is the fraction of defects that the inspection detected, as opposed to those that “escape” and are caught in later development phases, in tests, or in the field.2 Well-executed inspections should be able to find 60 to 80 percent of the life-cycle defects before the software goes into test. For example, in a project in which one of the authors was involved, inspections of the specifications, design, and code found 1,170 defects, while the various tests found only 825 defects.3 Mature software groups spend 10 to 20 percent of their resources on inspections. More information on inspections and their results is available elsewhere.1,4,5

References 1. T. Gilb and D. Graham, Software Inspections, Addison-Wesley, Reading, Mass., 1993. 2. W.S. Humphrey, A Discipline for Software Engineering, Addison-Wesley, Reading, Mass., 1995. 3. J. Rooijmans, H. Aerts, and M. Genuchten, “Software Quality in Consumer Electronics Products,” IEEE Software, vol. 13, no. 1, Jan. 1996, pp. 55–64. 4. M. Fagan, “Advances in Software Inspections,” IEEE Trans. Software Eng., vol. 12, no. 7, July 1986, pp. 741–755. 5. W.S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, Mass., 1989.

The GSS inspection process Baan piloted GSS support for inspections in 1996 and made it available to the engineers as an option in 1997. They could use it in the meeting room only, or in both preparation and the meeting room. We’ll call this GSS inspection to differentiate it from traditional paper inspection (where the scribe records on paper all the detected defects). GSS inspection’s main difference from paper inspection was that engineers electronically delivered the defects they found in preparation, either by email to the moderator or directly into the GSS from their desks. The scribe’s and moderator’s roles also changed. In a paper inspection, one of the inspectors typically was also the scribe. The GSS made the scribe redundant, relieving one engineer of a boring task. Previously, the moderator spent time preventing two people from speaking at the same time. However, GSS inspection allowed such communication because people May/June 2001

IEEE SOFTWARE

61

Group Support Systems GSSs, a form of groupware, have emerged over the past decade and increasingly are being used in a wide variety of business and government organizations. For example, GroupSystems is a GSS developed at the University of Arizona and subsequently commercialized by GroupSystems.com. It provides an integrated set of tools that groups use to generate information, dynamically share comments, classify and prioritize items, and perform a wide variety of other group support functions. Participants can interact both through computers linked on a network and verbally. GSSs (including GroupSystems) can also be used where all participants are not colocated in space or time. As such, GSSs become mechanisms for supporting and sustaining team activity in distributed organizational contexts. GSS use in organizations has focused on support for a variety of problemsolving and planning activities (for example, quality improvement teams) as well as for business process improvement.1 GSSs succeed by minimizing the process losses (for example, air time and participant apprehension) and seeking to maximize the process gains (for example, participant synergy) that impact groups.2 In a GSS setting, group dynamics play an important role in structuring and electronically supporting group tasks. For example, GSSs support anonymity when participants might be apprehensive about speaking their opinions in the presence of superiors. Organizations report dramatic time savings when using GSSs—for example, over 50 percent in people-hours and up to 90 percent in elapsed project time.3 These savings, coupled with high levels of effectiveness and participant satisfaction, have been replicated in a variety of business and government organizations.2 For more information, see www. cmi.arizona.edu, www.groupsystems.com, or www.groupsupport.com.

References 1. J. Nunamaker et al., “Electronic Meeting to Support Group Work,” Comm. ACM, vol. 34, no. 7, July 1991, pp. 40–61. 2. J. Nunamaker et al., “Electronic Meetings Systems: Ten Years of Lessons Learned,” J. MIS, vol. 13, no. 3, Winter 1996–1997, pp. 163–207. 3. D. Vogel et al., “Electronic Meeting System Experience at IBM,” J. MIS, vol. 6, no. 3, Winter 1990, pp. 25–43.

could “speak” via the keyboard. So, the moderator had more time for finding defects. Some moderators could allow limited discussion of major defects because the defect reporting did not require any talking. Preparation rates An inspection’s preparation rate (measured in lines of code per hour) significantly affects an inspection’s effectiveness and efficiency.8 Early in 1997, it became clear that some groups had increased their preparation rates at the expense of the effectiveness of the inspections. Consequently, proper preparation rates received more emphasis throughout the year. As a result, we could evaluate how the GSS performed at different inspection rates. We classified the inspections in terms of their preparation rates per engineer. A typical recommended preparation rate for source inspections is 200 lines of noncommented source code per hour.8,9 At Baan, the comment lines were also inspected, so we in62

IEEE SOFTWARE

May/June 2001

cluded them in the line counts. Therefore, the recommended inspection rate in this case was 200 to 300 lines of code. Inspections included in this study This study includes all the inspections of 4GL code in Baan’s Application Department during 1997. It excludes 3GL code inspections and all the inspections of designs, specifications, and other development documents. We excluded these inspections to facilitate a clean assessment of the impact of GSS use. Of the 4GL code inspections, we only considered those where a group of engineers inspected their own code. (Frequently, engineers from one development site inspect another site’s source code. This is especially important in an organization that is growing rapidly and where, consequently, the levels of experience are mixed.) These criteria let us compare the 87 GSS inspections to 102 paper inspections. Results We measured the results in terms of the major defects (majors) found. Figure 1 shows the effectiveness of the inspections in terms of majors detected per 1,000 lines of source code. Our strict definition of a major (see the “Inspections” sidebar) means that not all code changes that result from an inspection will be logged as majors. For example, improvements for performance’s or clarity’s sake are typically not logged as majors. We counted only those defects classified as majors after rework. We did not count false positives—that is, defects that were marked major but turned out to be minor problems (or not problems at all). Figure 1a gives the effectiveness of the inspections in a scattergram where a point represents every inspection. Figure 1b represents the same data in a bar chart. The chart distinguishes three classes of inspections: 200 to 300, 300 to 400, and 400 to 500 lines of code per hour. As we previously noted, the target preparation rate was 200 to 300 lines. Each of the six bars represents at least 20 inspections. Preparing and logging the 189 inspections took over 1,250 hours. The total meeting effort was 330 person-hours for the paper inspections and 261 person-hours for the GSS inspections. The numbers over the bars indicate the effectiveness of the inspections. We measured effectiveness as the weighted average for all inspections of that class—that is, the total number of majors divided by the to-

30 GSS Paper

Preparation rate The preparation rate is known to be a main independent variable for inspection quality. Table 1 shows the average preparation rates for the defined classes. As the table shows, differences in the preparation rate do not explain the differences in effectiveness. Software quality As we previously noted, all inspections concerned source code from the same major system. The engineers themselves decided whether to do a GSS inspection. The decision rested not on the source’s perceived quality but on more down-to-earth variables, such as the preference of the engineer

15 10 5 0 200

(a) 14

250 300 350 400 450 Preparation rate (lines per engineer per hour)

12.6

500

GSS Paper

12 Major defects per KLOC

Discussion The data indicate that the GSS inspections are more effective and efficient for inspections with preparation rates of 200 to 300 and 300 to 400 lines per hour. We believe that a GSS improves the inspection process by supporting it properly and enforcing it. For example, the requirement for inspectors to submit their defect reports before the meeting has encouraged them to contribute more to preparation. However, this is likely not only a tool technology issue. Because all other independent variables cannot be held constant in an industrial setting, other variables could have played a role. Two likely candidates are the preparation rate and the software’s quality.

20

10

9.0

8.5

8 6.6

5.8

6

6.6

4 2 0 200–300 300–400 400–500 Preparation rate (lines per engineer per hour)

(b)

Figure 1. The effectiveness of 87 GSS inspections and 102 paper inspections: (a) a scattergram; (b) a bar chart. .60

GSS Paper

0.52 Major defects per hour (preparation and logging)

tal size of the inspections of that class. The bar chart indicates that, for a rate of 200 to 300 lines of code, the GSS inspections find 40 percent more defects (12.6 over 9) per KLOC than do the paper inspections. The difference is 46 percent for 300 to 400 lines (8.5 over 5.8), but the difference is gone for 400 to 500 lines (6.6 for both the GSS and paper inspections). We define efficiency as the number of majors per person-hour for preparation and logging. We do not include the rework hours; we are interested mainly in the defect detection efficiency, and the GSS did not affect defect rework efficiency. Figure 2 gives the weighted average for efficiency. It indicates that the GSS inspections in the 200- to 300line range are 40 percent more efficient. The difference is only 20 percent in the 300 to 400 range. In the 400 to 500 range, the paper inspections are 17 percent more efficient.

Major defects per KLOC

25

.50 .40

0.44 0.37

0.46

0.39 0.36

.30 .20 .10 0 200–300 300–400 400–500 Preparation rate (lines per engineer per hour)

Figure 2. The efficiency of the inspections compared in Figure 1.

who moderated the inspection or the availability of the GSS facility. One useful measure of inspection results is yield (the percentage of detected defects—for more detail, see the “Inspections” sidebar). We do not have complete yield data for a number of reasons, including these: May/June 2001

IEEE SOFTWARE

63

Table 1 Average Preparation Rates for Inspections Preparation rates (LOC/hr.) Class GSS Paper

200 to 300 300 to 400 400 to 500

248 342 429

238 345 427

Difference in rate (percent)

Difference in effectiveness (percent)

4 −1 0

40 46 0

A GSS needs to fulfill additional requirements. Supporting software inspections was a new application for GSSs that identified opportunities and exposed weaknesses. On one hand, GSS functionality and architecture allowed quick response to new requirements. On the other, we identified these problems: ■

■

■

■

The defects found in early tests were not recorded at the same level of abstraction as was the inspection data. Our definition of a major makes it hard to compare the number of majors found to the number of test defects detected later. Modules were often not inspected as a whole but partially inspected and partially reviewed.

The limited yield data we do have suggest a higher detection yield for the GSS inspections. The code underwent a series of tests after the inspections. The data from the most extensive test show that the code that underwent paper inspection had 87 percent more fixes as a result of detected defects than did the code that underwent GSS inspection. Implications This study has implications for both GSS inspections and GSSs in general. GSS inspections We see three implications for GSS inspections. A GSS can contribute to more effective and efficient inspections. This implication is obvious but important, given that mature software groups spend significant time in inspections. Furthermore, it is significant because inspections are an important technique to assure quality software. Making inspection results available electronically might provide earlier and easier opportunities for improving software development. Distributed inspections are feasible. We do not have much data yet on the impact of distributing participants over different locations. However, our experience with inspectors using the GSS to handle preparation from their own desks encourages us to try distributed inspections. Given the international distribution of many development teams, this is an important opportunity. 64

IEEE SOFTWARE

May/June 2001

■

■

GSSs do not sufficiently support fixedformat input. The input in GSSs is typically free-format text. In some cases, it would be useful to fix the format and then ascertain, for example, that the second attribute is a one-digit attribute that allows distinguishing majors from minors. The GSS lacked integrated support for capturing metrics. For example, at a meeting’s end we want to know the number of majors and minors, and we want to accumulate such data across sessions and between groups. The GSS could not calculate this automatically. Poor interfacing between the GSS and Baan’s organizational systems limited expanded operational use. Seamless integration with the existing system and rapid, smooth movement of data between parts of an organization are paramount to operational success. This holds particularly for inspections because of their high frequency and because the participants are engineers. Engineers are very unforgiving if they repeatedly have to work around interface deficiencies. Integration with existing systems at Baan now lets engineers automatically export the sources that are to be inspected from the source code control system to the GSS. After the inspection, the meeting metrics are calculated and the detected defects are exported in the proper format to the defect-tracking system.

Research and development is underway to address some of these weaknesses. In particular, research projects are underway to improve GSS support for inspections, project tracking, and data modeling.6 For example, on the basis of the experiences we’ve described in this article, we are developing a GSS specifically for inspections. GSSs in general Software engineering is a particularly challenging application for GSSs. As we mentioned before, we are convinced that today’s

About the Authors Michiel van Genuchten is CEO of GroupSupport.com, a company that focuses on the development and application of group support systems. He received his MSc and PhD from the Eindhoven University of Technology. He has been involved with software process improvement and GSSs since 1991. He is a member of the IEEE. Contact him at GroupSupport.com, PO Box 80, 5600AB, Eindhoven, Netherlands; [email protected].

software problems might be tomorrow’s problems in other fields. So, we feel our results have implications for other GSS applications. Meeting protocol and tool support. The results show clearly that the benefits of GSS support deteriorate quickly when a proper process is not followed. In other words, when the inspection is not done well, it does not matter how it is supported. This is also a well-known conclusion from implementing information technology support in other fields. It is food for thought for those who have been trying to support meetings with a GSS. How effectively does a GSS support less mature meetings? How sure can we be, given that the outcome of meetings is hardly ever managed in quantitative terms? Increased attention to improving meeting processes and meeting metrics is warranted. GSSs in primary processes. The application of a GSS to support inspections was one of the first attempts to use a GSS routinely in daily work. Such use puts new and challenging requirements on GSS technology. We have labeled this use of a GSS as support for primary processes.10 Primary processes require “highfrequency” meetings under operational time pressure. In these meetings, participants know what they want to achieve and what is required from them. The meeting structure is mature, and consummating the process requires little to no special facilitation. Examples are the use of a GSS for emergency response or in the classroom for day-to-day teaching. The results of supporting inspections with a GSS give us high expectations for the application of GSSs in other areas.

Cor van Dijk is the manager of the Software Engineering Process Group of Baan Development. This group works on process improvements for specific units and for Baan Development worldwide. He studied mechanical engineering and has worked in software development in several different technical and managerial jobs for more than 16 years. He has worked on software process improvement since 1994. Under his guidance, Baan Development adopted CMM and PSP/TSP and has made significant process improvements. Contact him at Baan Development, PO Box 143, 3770 AC Barneveld, Netherlands; [email protected].

Henk Scholten is a software engineer in the Applications Department of Baan Develop-

ment. This department provides tools, ideas, and knowledge to help Baan Development write software efficiently. He studied electronics and has worked as a software engineer for many years. Contact him at Baan Development, PO Box 143, 3770 AC Barneveld, Netherlands; [email protected].

Doug Vogel is a professor of information systems at the City University of Hong Kong. He has been involved with computers and computer systems in various capacities for over 30 years, including being president of an electronics manufacturing company. He has been principally involved in the development and application of group support systems for the past 15 years. He received his PhD in management information systems from the University of Minnesota. Contact him at the City Univ. of Hong Kong, Dept. of IS, 83 Tat Chee Ave., Hong Kong; [email protected].

Acknowledgments We thank Baan’s engineers and moderators who participated in the inspections. We also thank the IEEE Software reviewers for their suggestions, which led to significant improvements of the article. Finally, we thank Tom Rodgers of Texas A&M University for his comments on earlier versions of this article.

References 1. 2.

3.

4.

T

he demands imposed on software engineering and its practitioners are ever increasing. Fortunately, combinations of methods, techniques, tools, and technology exist that will let us begin to address these demands. In particular, our findings open the door for broader consideration of GSS technology for software process improvement and suggest the development of process–technology hybrids. Many more opportunities exist to combine the best of the worlds of software process improvement and group support technologies. Both worlds will benefit as a result.

5.

6.

7.

8. 9. 10.

A. Porter and L. Votta, “What Makes Inspections Work?” IEEE Software, vol. 14, no. 6, Nov. 1997, pp. 99–102. J. Nunamaker et al., “Electronic Meetings Systems: Ten Years of Lessons Learned,” J. MIS, vol. 13, no. 3, Winter 1996–1997, pp. 163–207. J. Nunamaker et al., “Electronic Meeting to Support Group Work,” Comm. ACM, vol. 34, no. 7, July 1991, pp. 40–61. D. Vogel et al., “Electronic Meeting System Experience at IBM,” J. MIS, vol. 6, no. 3, Winter 1990, pp. 25–43. I. Tervonen, “Support for Quality-Based Design and Inspection,” IEEE Software, vol. 13, no. 1, Jan. 1996, pp. 44–54. T. Rodgers et al., “In Search of Theory and Tools to Support Code Inspections,” Proc. 31st Hawaii Int’l Conf. Systems Sciences, vol. 3, IEEE CS Press, Los Alamitos, Calif., 1998, pp. 370–378. M. Genuchten, W. Cornelissen, and C. van Dijk, “Supporting Inspections with an Electronic Meeting System,” J. MIS, vol. 14, no. 3, Winter 1997–1998, pp. 165–178. W.S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, Mass., 1989. W.S. Humphrey, A Discipline for Software Engineering, Addison-Wesley, Reading, Mass., 1995. M. Genuchten, J. Nunamaker, and D. Vogel, “Group Support Systems in Primary Processes,” Proc. 31st Hawaii Int’l Conf. Systems Science, vol. 1, IEEE CS Press, Los Alamitos, Calif., 1998, pp. 580–589.

May/June 2001

IEEE SOFTWARE

65

feature

iterative development

Heuristics for Iterative Software Development Drasko Sotirovski, Raytheon Systems Canada

s with many other important ideas, an iterative approach to software development is not exactly new (see the sidebar, “A Brief History”). Even at the time when the waterfall approach was mainstream, instincts led many a practitioner to follow the “fail fast” philosophy of iterative development.1 The following anecdotal quote is stereotypical:2

A The author discusses the foundation principles of iterative software development before proceeding to his main topic: a small set of useful heuristics for applying iterative development methods in practice. 66

IEEE SOFTWARE

According to the waterfall model we were using, the project was about halfway through the design phase. A lot of lower-level design and unit and integration testing remained. I was assigned one part of the system, and the remaining design and unit testing tasks were given to the 15 people we had allocated to us for the next five months. Given the amount of work that lay before us, these resources were not nearly enough. One subproject leader realized that our complex and lengthy design task would prove a huge challenge. His nose for risk analysis told him that our only hope was to try something new. So he gave a couple of us at the bottom a free hand to do the remaining project planning. Then he closed his office door and started looking for another job. I am still grateful to this man, because his lack of management involvement created a vacuum we were able to fill with bottom-up process improvements.

May/June 2001

We began by shifting to an incremental development model.

Practiced all along, often introduced by practitioners through the back door, iterative development methods are lately receiving their overdue formal recognition. And yet, very little material exists about this approach’s foundation principles. Most often, iterative development is defined as “not waterfall” without really stating the motivation and reasoning that justifies it. Understanding the motivation and reasoning at the heart of these methods uncovers a number of heuristics that are valuable in practice. This article is an attempt to fill in these gaps for software practitioners. It should provoke those who have successfully applied iterative development methods to ask themselves, what did we do right? Where did 0740-7459/01/$10.00 © 2001 IEEE

A Brief History

things go wrong? For others, it should provide advice for applying iterative software development in practice. Foundations of iterative development Engineering has so far produced two successful, accordingly accepted, and yet apparently opposite practices: ■

■

Do it right the first time. This practice is rooted in optimism and self-confidence with a firm belief that failing is not an option. In software engineering, this practice is best reflected in the waterfall model. Fail fast. This practice is rooted in an equally pessimistic belief that problems are inevitable. The question is not if we are going to slip but when and where. As far as software engineering goes, this practice is at the bottom of spiral and other iterative or incremental development models.

These opposing engineering practices might, at first sight, appear incompatible—iterative development and the waterfall approach are often described as if they have nothing in common. And yet, real-world projects (the solid line in the graph) are a mixture of these opposites, as Figure 1 depicts. Historically, engineering experience shows that increased uncertainty calls for iterative development. (The dotted part of the graph is only imaginary. If the probability of failure reaches certainty, the project is likely to not take off at all; on the opposite end of the spectrum, in the real world, there is always some probability to fail. In addition, the middle section of the curve is rather steep, showing the prevailing opinion that these two development methods have little in common.) Although these arguments might appear to be commonsense, they are surprisingly often neglected. The stereotypical complaint and most frequent criticism of iterative development techniques seem familiar: “An iterative development might find itself constrained by poor design in a previous iteration.”3 So true and yet so far from the mark! The whole purpose of iterative development is to find the weak points in the design of the previous iterations—and sooner rather than later. To give credit to the critics, iterative development techniques are not an excuse for doing sloppy design (as sometimes happens)

Iterative development has a long and interesting history going back at least to Frederick Brooks’ Mythical Man Month and Victor Basili and Albert Turner’s report at the first national conference on software engineering, both of which appeared in long-gone 1975. Basili and Turner looked on iterative development as a “practical means of applying stepwise refinement,” alluding to Niklaus Wirth’s famous article from even longer-gone 1971. Iterative development is certainly, together with software reuse, one of the recurring themes of software engineering. I would like to think that this holds a hidden lesson—that the history of iterative development appears to be iterative in its own right. Rediscovered again and again through several iterations, this development method matured over time and is now here to stay. And for good reason: as Bob Glass has said, quoting Ogden Nash, “At last I’ve found the secret that guarantees success: to err, and err, and err again, but less, and less, and less.”1 Reference 1. R.R. Glass, “Practical Programmer: Short-Term and Long-Term Remedies for Runaway Projects,” Comm. ACM, vol. 41, no. 7, July 1998, pp. 13–15.

simply because that sloppiness will be soon discovered. Its purpose is to discover pitfalls that otherwise could not have been foreseen. Do not mistake the presently increasing popularity of iterative development to mean that there is anything wrong with the “do it right the first time” model—quite the opposite. It is arguably a much more efficient (and less expensive) model, which is a tune dear to any manager’s ear. If there is anything wrong with this approach, it is that we just don’t always know how to. When we perform the same endeavor successfully again and again, we can argue that the “do it right the first time” model is the one to follow. But what if the engineering task is complex and novel? Is doing it right the first time possible? And if so, are we willing to bear the risks that stem from it? Sequential development processes critically depend on foreknowledge in system design and specification, which all too often does not match reality. Sequential development relies on designing Iterativeness and then building a large set of parts that at the end must perfectly fit into a system. For this purpose, designers need detailed foreknowledge on the way various pieces will be assembled and interact. Aye, there’s the rub: we often cannot attain this foreknowledge by anything short of building the sys-

Figure 1. Iterativeness and probability of failure (primarily driven by the foreknowledge in system design and specification) ought to be balanced.

May/June 2001

Probability of failure

IEEE SOFTWARE

67

Build first the system’s frame: bazaar-flexible where needed and cathedralstrict when necessary.

tem. Sequential development methods thus remain crucially misaligned with the intellectual difficulties inherent in system specification and design. Even though we can raise the bar for complex systems due to advances in technology, limitations inherent to humans will always make iterative development the method of choice for building novel highcomplexity systems. It should come as no surprise that some of the most exciting developments in software engineering, such as ex- treme programming,4 are still probing the limits of iterative development processes. Heuristics Designing software iteratively is often easier said than done.5 If the iterations are too small, iterating itself could consume more energy than designing the system. If too large, we might invest too much energy before finding out that the chosen direction is flawed. Design that appeared sound in an early iteration might show cracks later when it’s too late or too expensive to fix. The main dilemma is always the same: drawing a line in the quicksand between iterations. Developers seem to agree about each iterative cycle’s structure and workflow: from inception, through elaboration and construction, to transition.6 However, there is little advice on choosing the contents of the iterations. Tom Gilb offers a rare piece of guidance: “the juiciest one next” (meaning we should maximize the end user’s observed value of the system while minimizing our investment).7 Extreme programming similarly argues that the customer should decide the content of releases based on estimates the software shop provides. Both are sound pieces of advice, particularly when each iteration’s results are released to the end-user community. The ad -vice, however, has limited value when we need several iterations before the first release of the product to the customer. Even if planning the releases is entirely left to the customer, what other considerations should drive the customer’s decisions? Let’s examine a set of heuristics that I have found useful when applying iterative development methods to define the contents of subsequent increments. Understanding the heuristics helps provide vision and focus, which is essential when building complex systems. All my examples share the same do-

68

IEEE SOFTWARE

May/June 2001

main—that of an advanced air traffic control (ATC) system—admittedly simplified to illustrate the points for an audience that might not be familiar with the ATC domain. The choice is not accidental. Designers successfully used incremental development, and the heuristics described herein, to build the Canadian Automated Air Traffic System (see the sidebar), probably the most advanced system of its kind. A journey of a thousand miles starts with one step A design’s essence—its architecture—is contained in the relationships between the system’s different elements. Their harmonious cooperation is what makes or breaks the system. It is therefore important to assemble the architectural frame as early as possible, with only as much functionality as necessary to verify that the assembled frame is appropriate for the system. Having the frame in place will ■ ■ ■

at first, focus the effort; once integrated, boost confidence; and once understood, provide more insight into how the subsequent pieces should be put together.

So far, so good—but what does it mean to assemble the architectural frame? Researchers have found that one of the most prominent risks in building complex systems is excessive, immature, unrealistic, or unstable requirements.8 In other words, complex systems typically contain areas of volatile requirements, so it is paramount that such areas are identified early and provided with suitable bazaar-like architectural support, resilient to even dramatic changes in purpose or function. On the other hand, there are domains where functionality is well understood. The architecture can capture and enforce the needed patterns—such as the architecture of a cathedral does—thus making future iterations often more productive and virtually always more predictive. Functionality is not the only thing that evolves while developing a system. The computational environment (the hardware platform and the system software) is likely to change for any project that takes more than a year or two to complete; changes in scale (number of servers, workstations, and other components) and scope (demo installations, training installa-

tions, and so on) are other frequent contributors to the diversity that challenges projects. Looking from a different perspective, to reduce rework and mitigate risks in the areas where requirements and design are uncertain, we should precede bulk work on implementing large amounts of code with wide yet shallow integration that verifies the proposed framework. This approach encourages proactive management of the development effort by giving early attention to problems as they arise. Once the chassis is constructed, adding the parts usually carries less mutual dependencies; this is important for increasing parallel development in subsequent iterations. An ATC system, for example, deals with several subdomains with decreasing inherent stability: ■

■

■

■

the natural environment (the earth, the air space above it, the meteorological conditions, and so on), the aeronautical environment (navigational aids, airways, airports, runways, and so forth), flights (aircraft departure, reporting position, radar observation, arrival, emergencies, and so on), and monitoring and control (checking aircraft for conformance to clearances, correcting and improving estimates, distributing flight data to other air traffic service providers, and so forth).

Building the frame first means that early efforts must focus on establishing an infrastructure that supports the necessary interactions between the various application objects. For example, aircraft operators file flight plans, aircraft use aeronautical elements for navigation, aeronautical entities (when unavailable or out of tolerance) can influence the aircraft’s ability to follow a flight plan, aircraft and aeronautical entities are both submerged into the natural environment, and information about all of them are collected and, if significant, need to come to the attention of the controllers and neighboring air traffic service providers. Building the frame is building a wide, but shallow, representative sample of the total system functionality. It is, for example, a demonstration that a simple flight plan can be entered into the system, departed, progressed through a number of position reports, and arrived.

So, we should build the system’s frame (or skeleton) first. Look for structural breakage—for example, the frame should be flexible when needed and stiff where necessary—and correct it before proceeding. Doing it later will certainly be more expensive, if not too late. Though the Way is quite broad, people love shortcuts One of the essential motives for iterative development is to reduce risk by validating the proposed design as early as possible. Validating means different things to different people, yet for an impartial judge of a complex system, only the functionality truly integrated into a system counts. Anything less than full integration goes against the very foundation of iterative development: We should always keep in mind that the intent is to discover problems that otherwise could not be predicted, and if all pieces are not integrated into a coherent system, design flaws might remain hidden. Taking shortcuts is acceptable—for prototyping. And prototyping, in any other but the loosest sense of the word, should not be confused with iterative design and development of a complex software system. Prototyping is focused on proof of a concept with an implicit or even explicit intent to throw away the artifact once it has served its purpose. Iterative design and development of a complex software system has no throwaway contents. Full design attention is paid to all aspects of the system, and mandatory integration at the end of each iteration is the ultimate impartial judge of the design and building efforts. Unlike in prototypes, throwaway happens by accident or, to be more precise, because of the designer’s inability to foresee all the subtle relationships between the many different elements of a complex system. But it’s not by intent. Often, an iteration is not fully integrated. Lack of loyalty to the foundation principles together with lack of time are the main contributors to unbalanced and incoherent iterations. A typical excuse is “nothing can go wrong there,” forgetting that iterative development was chosen for exactly the opposite reasons: to discover breakage at unsuspected places. This excuse is often accompanied with “it’s time to move forward,” a Dogbertism for “we spent enough

Loose ends and shortcuts only deprive you of an early discovery.

May/June 2001

IEEE SOFTWARE

69

Canadian Automated Air Traffic System In the mid 1980s, Transport Canada, NAV Canada’s predecessor, which operated the Canadian Air Navigation System (ANS) at the time, embarked on a process to upgrade or replace several of its existing systems with a modern, highly integrated air traffic management system. This new system was called the Canadian Automated Air Traffic System. CAATS would automate many existing manual processes and integrate isolated systems. With safety paramount, CAATS was designed to relieve controllers of many of the low-value, manual processes they had to perform, while adding a number of advanced automated features that did not exist in current ATC systems. In 1989, Transport Canada awarded Hughes Aircraft of Canada Limited (the predecessor of Raytheon Systems Canada) the contract to design, develop, and deliver CAATS. CAATS started as a traditional waterfall program. Beset with the usual difficulties of many software-intensive projects (such as unstable requirements and rapidly changing technological capabilities) during the mid 1990s, we restructured the CAATS program into an iterative incremental endeavor, which rapidly started to gain momentum. Over the past eight years, as many as 200 software developers have written almost 2 million lines of Ada code. These efforts culminated in the

recent successful completion of the final Software Integration Test and Factory Acceptance Test. This final CAATS milestone is the climax of the previous six milestones achieved at or ahead of schedule. CAATS provides a full range of ATC functions for the seven Area Control Centers and 23 Tower Computer Complexes that control the entire Canadian civilian domestic airspace. CAATS offers a complete system solution containing all air traffic management and support functions, including the roles of air traffic controller, system manager, data controller, supervisor, and operations support. In addition, a number of external interfaces are integrated into the system to support flight information exchange and atmospheric and aeronautical data acquisition and application. CAATS handles International Civil Aviation Organization, domestic IFR (Instrumental Flight Rules), and VFR (Visual Flight Rules) flight plans. The system provides a rich tool set for controlling aircraft both in and out of radar coverage. Controllers can also quickly enter and modify detailed IFR flight clearances. Using adapted aircraft performance models, adapted route information, and current wind data, CAATS transforms clearances into highly accurate four-dimensional profiles. This distinctive feature, in turn, supports system functions of automatic flight data posting, conformance monitoring, hands-

money and energy, so let’s abandon the remaining work—no one will notice and we’ll do it in the next iteration.” Whenever this happens, be warned: it is more likely to backfire than not. (If it doesn’t, it means that we should have picked the “do it right the first time” model!) As an example, a truly end-to-end integrated ATC system often shows, on closer inspection, unanticipated breadth. Even relatively simple entities such as airways comprise a number of mutually dependent functional elements: ■ ■

70

IEEE SOFTWARE

May/June 2001

a means to prepare data about airways offline, mechanisms to use the prepared airway

■ ■ ■

off warnings, adapted automatic hands-off and acceptance, and flight data distribution. CAATS builds four-dimensional VFR profiles as well, which provide a basis for VFR automation, such as late arrival monitoring. Finally, CAATS includes a conflict prediction capability that evaluates IFR clearances against separation rules (spacing between aircraft). Conflict probe capability allows a controller to “what-if” a clearance before committing to that clearance. The controller can convert planned clearances to current clearances and forward them to flight service stations or towers as required (see Figure A). CAATS can consolidate or deconsolidate master sectors of airspace under a single controller as traffic levels vary. Sectors can be dynamically configured to use multiple adjacent workstation display screens for one or more controllers. User and sector preferences combine to facilitate easy setup of the many adaptable display objects. The controller workstation displays any combination of situation and tabular data. The situation display comprises an integration of surveillance, weather, and flight data over an adaptable multilayer color map. The display is highly interactive, allowing the controller to immediately access and modify flight data, obtain current status data on map objects such as airports, airways, restricted areas, and special-use airspace.

data or to migrate from one version of the data to the next, a means to designate if an airway becomes unavailable, use of the data to analyze filed flight routes, and showing the layout of the airway structure at situation displays.

This heuristic emphasizes the importance of first understanding what “end to end” means, defining the implementation depth for the various bits and pieces, and finally (and most importantly) integrating the full breadth of it. The implementation depth, particularly in early iterations, can be shallow, but we should never sacrifice breadth. More impor-

Tower

FDP/SDP Tower workstations

Radar FDP/SDP Control workstations communications gateway Area control center

Terminal Control Unit

Military Terminal Control Unit

Area Control Center

Tech center

WAN

Area control center

AFTN users

Weather NOTAM Traffic Foreign Remote office office Management centers strip printers

Flight Airport Time service sensors source stations

Air defense

Rescue coordination center

Figure A. CAATS integrates adapted and acquired data about the natural environment, the aeronautical environment, and current and planned flights to support a number of advanced automated features.

tantly, the iteration, for various reasons, might fall short of implementing all the planned depth—yet we should never surrender the full integration. Even if offline preparation of the airway data is limited to just a few of the many airway attributes or the flight route analysis performs only fundamental referential integrity checks, the full end-to-end integration remains the only factual evidence of progress. So we should plan to integrate the system end to end. And do it. Integration will prove, or disprove, what would otherwise only remain a claim. Loose ends and shortcuts only deprive you of an early discovery. Impartiality is kingship A complex system is the result of a large

team’s collaborative effort. Different team members contribute their expertise to different aspects of the system. Because of their different know-how, they have different opinions about the functionality, which must be built early to mitigate risk. There is only so much we can squeeze into the next iteration, so what should we do? How do we pick and choose? Despite the expertise, intuition, and reviews, many of the problems uncovered during integration are not in any of the areas that different experts worried about. This should not come as a surprise. First, iterative development was selected exactly because of the system’s unknowns and inherent complexities, suggesting that the human mind might not anticipate all possible pitfalls. May/June 2001

IEEE SOFTWARE

71

Pay attention to a system’s observed weaknesses, not the perceived ones.

Why should we then be surprised when integration uncovers design flaws at unexpected places? Second, expert intuition focuses the design attention to the expected trouble spots, and, owing to all the attention, the resulting design is often flawless—cracks show elsewhere. Consequently, in iterative development it is important to grow the functionality uniformly across the system. Different human experts will have different aspects of the system dear to their hearts, but overall development, to be successful, must remain impartial so that no aspect is neglected. In every system, there is some mainstream functionality considered to be its essence. In an ATC system, for example, processing flight plans, issuing clearances to aircraft, predicting future conflicts, and so forth are the star functionalities. An ATC system, however, contains a surprising number of additional elements: ■ ■ ■ ■ ■

a means to prepare data about aeronautical infrastructure, mechanisms to make the system use the prepared data, provisions for monitoring and managing the system, switch-over and other elements of faulttolerant behavior, and capabilities to record and then play back the system’s execution for incident analysis.

These functionalities, although essential for system operation, are sometimes neglected simply because they live in the celebrity’s shadow. Not just functionality but capacity might also be neglected. And yet, if the system cannot scale up to, say, the required number of workstations and servers, it is just as broken as when the flagship flight plan processing breaks. It sounds like common sense, but it is exactly what a number of recent ATC programs have reported: the Advanced Air-Traffic System in the US and the New En Route Centre in the UK. So we should grow the systems, over iterations, in an impartial and balanced manner such that all aspects of the system receive due attention. Adjust the attention based on the system’s observed weaknesses, not on perceived ones. 72

IEEE SOFTWARE

May/June 2001

What do Heaven and Earth rely on to be old and enduring? No matter how thorough the testing, complex software gains over time a special quality—it matures. In particular, as the number of residual software faults decreases over time, the software becomes more and more reliable. Although there might be methods (such as fault injection) that artificially age the software, the maturing effects of the natural aging process are hard to match. Moreover, not all the software that makes up a complex system is equal. First, not all functionality is of the same importance for achieving a system’s mission, which makes a residual fault sometimes easier and sometimes more difficult to tolerate. Second, all the code is not executed with the same frequency—the program counter often spends a large percentage of time repeatedly looping through small portions of the code, which again makes some residual faults easier to tolerate than others. For these reasons, it is important to build the mission-critical pieces of the software early on—as well as the pieces that the program counter will visit most often. Even if the design is relatively simple and straightforward, this will ensure over iterations a maturity level that is otherwise difficult to achieve. A software component that lives through multiple iterations gains in a number of other important ways. From iteration to iteration, as the clients of the software component evolve and change, they expose the component to different usage patterns, thus stressing the underlying design and making hidden flaws surface. This has a purifying effect on the component’s design and often results in the design becoming simpler over time. It also ensures that the component’s design does not rely on any particular usage pattern, thus facilitating the system’s future maintenance. Furthermore, the software that executes the most also contributes the most to overall system performance and responsiveness. When developed early, software components with high frequency of execution will also expose their performance properties early and provide ample lead time for (if need be) multiple optimizations. But what if the critical functionality that ought to be given a chance to mature is exactly the functionality where the requirements are particularly unstable? If this is the case, early iterations should focus not on im-

plementing the requirements but on implementing a framework resilient to requirements changes. If the requirements are indeed as unstable as foreseen, when the requirements mature over subsequent iterations, the framework’s resilience to change will be put to the test, stressed, and matured. In an ATC system, for example, there is hardly anything more important than timely and accurate use of the data received from radar sources: correlating the radar returns with flight plans and clearances, deriving departure and arrival messages, monitoring flights’ progress and their conformance to current clearances, and so on. Because an ATC system receives new observations with every sweep of the radar antenna, processing radar data is also the most frequent task it performs. Although the underlying functionality is well known and hardly novel, it is still worthwhile to build the radar data processing early and let it mature iteratively to perfection. However, advanced ATC automation, in particular features not present in existing systems and therefore not proven in practice, are a major source of requirements instability. For that purpose, building a framework early for implementing automation agents will ensure that the volatile requirements are not using a brittle infrastructure. So we should build the mission-critical functionality early. Early functionality has more time to mature and gain the quality properties that come with age.

T

here is a sign that software engineering is coming of age: the search for the silver bullet, the Holy Grail of Software, is being replaced with thoughtful application of many accumulated best practices. Among the best practices, iterative software development models are themselves coming of age: from discovering the critical misalignment between the reality of software development and its models, over attempts to change reality to fit the models, to finally embracing reality, and brave new—even extreme—ideas for the future. Applying iterative development in practice still faces many obstacles: buy-in from various stakeholders, organizational structure, and experienced personnel, to mention just a few. Among them, outlining the contents of the

subsequent increments is a technical challenge that can both augment and diminish the benefits of iterative development. This article offers several heuristics for defining the contents of subsequent increments—heuristics that promise to have a wider area of applicability than the real-time control and simulation they’ve been proven in. One final warning: heuristics are just what they are—vague, speculative, difficult to quantify, but valuable when applied with care.

Give the mission-critical functionality time to mature.

Acknowledgments I thank Alex Bell and Philippe Kruchten for their patience with the early versions of this article. Most section titles are quotes from Tao Te Ching (see www.acmuller.gol.com/laotzu). Eric S. Raymond introduced the cathedral and the bazaar metaphor, although for a different purpose—see http:// sagan.earthspace.net/esr/writings/cathedral-bazaar.

References 1. B. Curtis, H. Krasner, and N. Iscoe, “A Field Study of the Software Design Process for Large Systems,” Comm. ACM, vol. 31, no. 11, Nov. 1988, pp. 1268–1287. 2. A.B. Jakobsen, “Bottom-up Process Improvement Tricks,” IEEE Software, vol. 15, no. 1, Jan./Feb. 1998, pp. 64–68. 3. R.N. Charette, K.M. Adams, and M.B. White, “Managing Risk in Software Maintenance,” IEEE Software, vol. 14, no. 3, May/June 1997, pp. 43–50. 4. K. Beck, “Embracing Change with Extreme Programming,” Computer, vol. 32, no. 10, Oct. 1999, pp. 70–77. 5. P. Kruchten, “From Waterfall to Iterative Lifecycle: A Tough Transition for Project Managers,” The Rational Edge, Dec. 2000, www.therationaledge.com/content/ dec_00/m_iterative.html (current 18 Apr. 2001). 6. W. Royce, Software Project Management: A Unified Framework, Addison-Wesley, Reading, Mass., 1998. 7. T. Gilb, Principles of Software Engineering Management, Addison-Wesley, Reading, Mass., 1988. 8. E.H. Conrow and P.S. Shishido, “Implementing Risk Management on Software-Intensive Projects,” IEEE Software, vol. 14, no. 3, May/June 1997, pp. 83–89.

About the Author Drasko Sotirovski is a software architect at Raytheon Systems Canada. He has 20 years’ experience in developing large-scale real-time software for defense, simulation, transport, and telecommunication systems for several European and North American customers. His research interests are software architecture and distributed object-oriented technologies. He received a BSc in technical physics and computer science from Elektrotehnicki Fakultet u Beogradu, Yugoslavia. He is a member of the IEEE Computer Society and the ACM. Contact him at Raytheon Systems Canada Ltd., 13951 Bridgeport Rd., Richmond, BC, V6V 1J6, Canada; [email protected].

May/June 2001

IEEE SOFTWARE

73

feature software process

Defining Team Processes Using OO Metaphors Csaba Egyhazy, Virginia Polytechnic Institute and State University Scott Eyestone, Battelle Memorial Institute Janet Martino, US Air Force

For a large, complicated project, this team developed a technique for process definition using the metaphors of object-oriented technology. The OO approach allowed a detailed work breakdown for a dauntingly complex set of interrelated responsibilities and collaborations. 74

IEEE SOFTWARE

n 1996, the Office of the US Assistant Secretary of Defense (Health Affairs), Clinical Business Area, began investigating the use of objectoriented technology for its Computer-Based Patient Record program. It commissioned the CIOOT project (CPR Interoperability Using Object-Oriented Technology) and tasked multiple vendors and government offices to work within an integrated product team.

I

At first, bringing order to the many diverse and conflicting views represented on the team seemed as impossible as herding cats. To deal with the project’s complexity, the CIOOT project management team developed a process definition technique based on the metaphors of OO technology. The technique brought about common understanding and a clear project plan and, as a bonus, proved valuable for many other practical purposes. Just getting started After an initial demonstration of OO analysis and design techniques,1 the CIOOT team’s task was to develop a proof-of-concept implementation demonstrating that open, distributed-object computing could achieve interoperability among heteroge-

May/June 2001

neous, automated information systems. The implementation they chose was a standardscompliant architecture for the US Department of Defense Military Health System. The MHS has been called the world’s largest HMO that occasionally goes to war. An architecture for it would require complex interoperability across many military and civil organizations, each with widely disparate functions and information systems. The complexity of the health care subject matter, the special domain of military health care, and the frequency of changes in business practices all added to the challenge. The interoperability infrastructure would have to be secure, reliable, maintainable, scalable, ubiquitous, and seamless to users. In short, it would have to demonstrate adaptability. To achieve this 0740-7459/01/$10.00 © 2001 IEEE

objective, the CIOOT team chose a Common Object Request Broker Architecture (Corba) middleware approach.2 With a rough idea of what to build and a charge to start building a piece of it, the CIOOT team had to decide how to build it. At the time—1996—the literature on OO technology was much less robust; there were no cookbooks. Magnitude, complexity, and funding made an incremental approach necessary. A chaotic array of documents, capabilities, government organizations, and vendors were potentially relevant; the CIOOT team management had to figure out how to simplify complexity and bring order to the chaos. So the initial objective was to clearly assign responsibilities and deliverables. Fortunately, team members accepted the assertion of the Software Engineering Institute’s Capability Maturity Model that process definition and continuous process improvement constitute a necessary risk mitigation activity for such a situation.3 The team’s managers and technical leadership wrestled with several techniques to plan and organize work. However, the magnitude and complexity of the tasks were too great to directly define a work-breakdown structure in Gantt chart form. The problem’s complexity and the group’s relative inexperience with the object paradigm caused the unceremonious erasure of many whiteboard drawings. During one session, however, a key hypothesis emerged from the team’s technical leadership: Since OO technology is particularly effective at simplifying and bringing order to software complexity and chaos, why not apply its principles and analytical techniques to process definition? Many team members had recently read about and were intuitively drawn to David Taylor’s convergence engineering4 and Technium’s class-based reengineering concept.5 The CIOOT team proposed to extend these concepts, defining a software development process by specifying the necessary responsibilities and collaborations among the providers of common project services and the developers of the project’s architectural element products. Common project services included analysis, design, requirements management, and configuration management. Architectural

elements included legacy system wrappers and many of the components of Corba middleware and services. The CIOOT group came to perceive these services and element production as logical work units that they could define as a “process class.” With faith in its reasoning but insufficient resources for a control group, the CIOOT team proceeded, employing diverse “best-of-breed” techniques and process elements from multiple sources. Ambler’s description of Class/Responsibility/Collaboration (CRC) cards, for example, provided a strong influence.2 In addition, elements of the Corba target architecture and the emerging literature on the process for implementing it provided insight toward the work units needed. For instance, the team knew that a legacy system would have to be exposed in object form and that it should show interoperability with a commercial system. This requirement implied a work unit to develop each system’s interface to the middleware. Similarly, the probability of nonstandard data among the participating systems implied the need for a semantic mediator, or lexicon service. Knowledge of business practices in the health-care domain implied the need for order entry, results retrieval, and resource access-control services. The literature of the time emphasized the importance of OO analysis and design, so these were also logical work units. The team considered the theoretical logical work units as classes, defined class responsibilities as tasks with level-of-effort parameters, and articulated collaborations as explicit interface definitions expressed in terms of dependencies and products, with definitions articulated as acceptance criteria. Once the team started thinking in those terms, the process structure started taking shape quickly. At the time, the tools available were a whiteboard, PowerPoint, and Microsoft Project.

Since OO technology is particularly effective at simplifying and bringing order to software complexity and chaos, why not apply OO principles and analytical techniques to process definition?

Actions Team management began by asking the members of each anticipated logical work unit to write down their own responsibilities and collaborations in general CRC card form. Responsibilities were to include level-of-effort estimates written as full-time staff equivalents over a time period expressed in months. Definitions of collaborations were to include the May/June 2001

IEEE SOFTWARE

75

Functional Requirement

1.“Shall” statements 3. Collaborate; FR capture

1.“Shall” statements OO analysis

Requirements management 11. Analysis UC to FR trace

6. Draft Analysis UCs

18. Faceted components to design class trace

2. Review/system 9. Validated UCs familiarization 3. In collaboration with RQM, aggregate shall 5. Collaborate statements, develop scenario story diagram 4. Prepare and present mission analysis briefing 5. Incorporate “unaggregated” shall statements as rules/ constraints/mechanisms in UCs 13. Faceted 8. Validate scenarios, roles, components screens and UCs via SME workshop 11. RQM collab; trace OOA UCs to FRs 12. Prepare and conduct analysis review. Freeze functional 9. Validated UCs baseline.

Key UC: FR: BR: SME: RQM:

Use case Functional requirement Business rule Subject matter expert Requirements management

Configuration management

Reference model

22. Design UC to Analysis UC trace

16. Design UCs

2a. Review HC models/standards 13. Faceted components 3. Identify architecture OO constraints/enablers design 5a. Collaborate with OOA 7. Review draft Analysis UCs 14. Review faceted components, 10. Review validated UCs legacy interface BRs, analysis 11a. Select “best of breed” models/ UCs, and architecture inputs standards and translate 15. Assign BRs/events to classes to best constructs to UML develop Design UCs 12a. Formulate “faceted” (start sequence diagram) components and place 17a. Associate info items with class in container attributes & operation parameters; 17. Model relationships formulate classes using “gold between faceted standard,” “working data,” and components and design UCs legacy system definition 18. Compile traceability 18a. Build dynamic object model matrix-faceted 19. Document OOD reuse components to 20. Publish Rose model glossary OO Design classes per design 16. Design UCs 22. Compile traceability matrices, design packages to Design UCs 21. Design model to Analysis UCs 23. Prepare for design review

Figure 1. A brainstorming session might produce this kind of informal spider diagram.

specific products required as input and the products they would produce as output. Next, project management asked each potential logical work unit team to develop CRC cards for each of its “process neighbors”—that is, for each collaborator from whom they expected to receive services or products and for each collaborator to whom they expected to provide services or products. The third step consisted of planning sessions in which the stewards of a potential architectural element or service met to hammer out details of collaboration with its process neighbors. The CRC cards from the first two steps stimulated a considerable amount of dialogue and quickly exposed redundancies, gaps, and faulty assumptions in tasks and interface products. (For instance, the interface product issues included defining the nature, format, and distribution of the products.) 76

IEEE SOFTWARE

May/June 2001

These sessions usually began with a team member drawing the entity being considered in the center of a copyboard and then drawing curved collaboration lines to other process classes. Thus, the term “spider diagram” came into the team’s parlance. Figure 1 shows the PowerPoint documentation of a copyboard spider diagram for three logical work units: reference modeling, OO analysis, and OO design. The outlined box indicates the central entity—in this case, reference modeling. For simplicity and space considerations, Figure 1 is an aggregated extract of the actual diagrams developed for these three services. As they brainstormed a spider diagram, participants approximated the sequence of tasks and products. They also began assessing the level of effort necessary to perform the work, including time for planning, meet-

ings, educational activities, and coordinating with other work units. (The time allotted for such coordinating activities became informally known as the “collaboration tax”—the level of effort required to maintain situation awareness and common understanding among all team members. We’ll return to this later.) In the fourth step, the CIOOT team compiled spider diagrams to depict the total process necessary to accomplish an increment. (Each increment is a discrete function within the health-care domain, such as patient registration or results retrieval, or a domain service such a particular person or lexicon.) Figure 2 is a very simplified view of the compiled spider diagrams, omitting the responsibilities and collaboration products for each process class. It depicts an introductory overview of the process classes and their relationships to each other. Note the configuration and requirements management tasks stretching across the full project life cycle. Although the CIOOT team did not have the resources to fulfill SEI CMM Level 3 to the letter, we were determined to practice strong configuration and requirements management discipline to approach its spirit. The red dotted box highlights the three work units we focused on in Figure 1 and will again focus on in Figure 3, the detailed process diagram. The fifth step consisted of a face-to-face validation conference attended by the entire team. The group spent three arduous days refining all tasks, dependencies, sequences, and product interfaces. Figure 3 is an extract from an actual process diagram that emerged from that conference, simplified to ease understanding. Actual process diagrams contained more detail than we could present here; each also had back-up documentation such as detailed product definitions in the form of acceptance criteria. Although simplified, Figure 3 depicts key fundamental process patterns and documentation conventions that emerged: ■

In general, work units did not send products directly to the process classes that needed them. They placed them under configuration management control, which, in turn, alerted work units of the product’s availability. The receiving

■

■

■

■

work unit then checked the product out from CM. Process classes have responsibilities listed under their name box. Process classes that were essential collaborators but external to CIOOT funding (none shown in Figure 3) appeared in boxes with dotted outlines. Process classes not active for a particular mission increment (none shown in Figure 3) were grayed out. Solid arrows denote the flow of required products into a process class and the outward flow of products developed from the process class. Dashed, two-headed arrows indicate informal collaborations for preproduct interchange. The product acts as the formal interface between teams. The alphanumeric identifier of each responsibility and product indicates its process class and approximate relative occurrence in time. The alpha portion of the identifier suggests the work unit and the numeric portion is an indicator of relative time. For example, FR1 means functional requirement elicitation, first responsibility to be executed. In Figure 3, task A22 in OO analysis and RFM22

Requirements capture

Requirements management

Reference model Requirements elicitation

OO analysis

Architecture

OO design

Middleware implementation

Server system wrapping

Testing

Client system interface

Configuration management

Figure 2. A simplified overview of process classes. May/June 2001

IEEE SOFTWARE

77

Requirements management RQM12. Capture

RQM25. RQM27. Collab with Review traceability OOA to relate matrix, correct AUCs to FRs shortfalls with (2FTEs x .25 mo.) OOA (2FTEs x .25 mo.)

RQM49. Collab with OOA to trace faceted (2FTEs x .25 mo)

A26. Initial traceability matrix– AUCs to FRs (Requisite Pro)

CM11. CM30.

(email) –OOD products (email)

D38. Design UCs RFM16. Review HC models/standards (2 FTEs x .5 mo.)

In collaboration with RQM, aggregate shall statements,

Identify architect constraints/ enablers (2 FTEs x .5 mo.)

(3 FTEs x 1.5 mos.) A20.

Collaborate with OOA; review analysis UCs (2 FTEs x .75 mo.)

A22. “

” shall

.5 mo.) A23.

Alert– t FR t– documents (email) Alert– t–AUC t–

(2FTEs x .25 mo.)

model

A16. Review/system familiarization (3 FTEs x 5 days)

UC trace (RP) # RQM51. Final faceted component to Design class trace (RP) #

matrix, correct

RQM44. RQM45. Collab with OOD Review traceability to relate pkgs matrix, correct shortfalls with OOD (2FTEs x .25 mo.) (2 FTEs x .25 mo.)

RFM49a. Initial traceability matrix– fac. components to design UCs (RP)

OO analysis

RQM13. FR only file (RP) RQM28. Final FR/AUC trace (RP) # RQM46. Final design

RQM50.

A25. RQM collab; trace OOA UCs to FRs (3 FTEs x .25 mo.) A27. Refine UC to FR trace with RQM (3 FTEs x .25 mo.)

Select “best of breed” models/ standards and translate best constructs to UML (2 FTEs x 1.5 mos.) RFM29. Formulate “faceted” components and place in container (2 FTEs x 1.5 mo.) RFM39. Model relationships between faceted components and design UCs (2 FTEs x .5 mo.) RFM49. Compile traceability matrix— faceted components to OO Design classes (2 FTEs x .5 mo.)

SWT. Prepare and conduct Analysis Review. Freeze RFM30. Faceted components (Rose, UML) #

(3 FTEs x .5 mo.) A21. Mission analysis briefing (PowerPoint) # A24. Validated OOA Scenarios and Analysis UCs (MS Word) # A31. Analysis review #

#: Deliverable to the government

CM11. Alert–FR documents

CMn .

CM15. Alert–“FR only”

CM26. CM48a-e.

Alert–Arch. – docs (email) –

OO design

D36. Review faceted components, legacy interface BRs, analysis UCs, and architecture inputs (3FTEs x .75 mo.) D37. Assign BRs/events to classes to develop Design UCs (start sequence diagram) D39. Associate info items with class attributes classes using “gold standard,” “working data,” (3 FTEs x 1 mo.) D40. Build dynamic object model (3 FTEs x .5 mo) D42. Document OOD reuse (3 FTEs x .25 mo.) D43. Publish Rose Model Glossary per design (3 FTEs x .5 mo.) D44. Compile traceability matrices—design packages to Design UCs to Analysis UCs (3 FTEs x .25 mo.) SWT. Prepare and conduct Design Review (3 FTEs x .5 mo.) D46a. Implementation design model* (Rose 4.0) # D46b. Rose Model Glossary (MS Word) # D46c. OOD report; includes reuse rpt (MS Word) # D46d. Traceability matrices—Design package to UCs # D46e. Design review (PowerPoint bfg) # CM11. Alert–FR documents (email) CM30. Alert–AUC – / FR Trace (email) CM33. Alert–Functional baseline freeze (email)

–

Detailed SW design document

Configuration management CM14. Iterate 6-10 (FR only RP file) CM47d. Iterate 6-10 (Design pkg to DUC trace) CM51f. Iterate 6-10 (Faceted component to design class trace)

CM6. CM7. Register item CM9. CI availability CM10. Maintain CI control

CM25. Iterate 6-10 (Valid UCs) CM31. CM31a. Iterate 6-10 (Fac. Comps.) CM32. SWT CHCS II PM IPR-set priorities and freeze functional baseline

CM32a. Iterate 6-10 (BRs) CM47a. Iterate 6-10 (design model) CM47b. Iterate 6-10 (glossary) CM47c. Iterate 6-10 (OOD rpt.) CM47e. Iterate 6-10 (design review)

Figure 3. Process diagram extract. BR indicates business rule; FR, functional requirement; HC, health care; RP, Requisite Pro, a computer-aided system engineering tool that traces requirements throughout the system life cycle; SME, subject matter expert; SWT, structured walk-through; UC, use case. 78

IEEE SOFTWARE

May/June 2001

■ ■

■

■

in reference model are collaborative actions that occur in approximately the same time space. The # symbol indicates deliverables to the government. The parenthetical expression following a responsibility provides the estimated level of effort for that subtask. For example, “3 FTEs × 1.5 mos” means 4.5 staff-months of effort. The parenthetical expression after a collaboration product specifies the product’s format. For example, “Rose, UML” means the object model would be delivered in Rational Rose file format in Unified Modeling Language notation. Some products were key artifacts to requirements tracing within the requirements management process class. An artifact’s developer performed the trace in collaboration with the requirements manager.

For the first increment, Patient Registration, we ported the responsibilities of each process class to a work-breakdown structure in a Gantt chart with minimal changes. The Gantt charts showed products as milestones; the process class diagram was simply a project plan in another form. For the second work increment, Results Retrieval, we tailored the process class diagram to the new assignment. We used the responsibilities to quickly generate contract task statements and the products to develop deliverables lists. Results As this technique unfolded, we quickly saw that it readily exposed misconceptions, faulty assumptions, inexperience with OO technology, and biases. Fortunately, the target architectural concept was sufficiently clear to keep process definition on course. Even with the notion of what to build in mind, anticipated logical work units changed as the CIOOT team worked through the details of how to build it. The process class work teams soon discovered that the technique highlighted dependencies and that these dependencies had to be expressed in terms of formatted products, with definitions and acceptance criteria agreed upon by senders and receivers. As in software development, the highest risk lay in

interfaces, or borderline conditions. Our OO technique flushed out collaboration interface issues such as gaps, the precise nature of products, product distribution, and redundancies. Despite understanding the risks of interfaces and making an extra effort to make them explicit during process definition, some products were initially ill-suited to recipients’ needs. The CIOOT team’s informal collaboration practice of deploying downstream recipients of products to work with upstream producers generally kept this potential problem in check. The approach also helped us identify the artifacts we should place under CM control and put into the requirements management life-cycle trace. Some team members were new to the CM and RM disciplines in an OO environment. Clear CM and RM process definition fostered understanding and eased anxiety. It also contributed precise requirements for selecting CASE (computer-aided software engineering) tools. Initially, some team members resisted the high level of effort needed to work through this technique. It consumed nearly all of the team for two workweeks. By the end of that period, however, the team realized that this hard work had simplified the project’s complexity, and that our detailed, common understanding of responsibilities and collaborations would substantially mitigate risk and save time and effort in the next work increment. Our technique had made responsibilities, assumptions, and interdependent collaborations explicit. An unanticipated benefit of our process definition technique was its effect on team management. Because they had a stake in the outcome, managers listened carefully to higher-level technical discussions regarding tasks and their associated levels of effort. They asked questions and provided input from the managerial perspective that led to enhanced understanding between the management and technical communities. Managers and technical newcomers also learned the fundamental principles of OO technology. This paid substantial labor and management dividends as the project progressed over multiple increments. On the negative side, the tools we used to support our technique were cumbersome. Porting information from copyboard to PowerPoint to Microsoft Project was invari-

As the technique unfolded, it readily exposed misconceptions, faulty assumptions, inexperience with OO technology, and biases.

May/June 2001

IEEE SOFTWARE

79

The first increment was successful and fell within cost and schedule tolerances, depite the project’s complexity and the team’s relative inexperience with OO technology.

80

IEEE SOFTWARE

ably manual. Documenting change was laborious. Any change in task sequence numbering triggered tedious effort. Thus, as the project began, encapsulating the process class responsibilities proved valuable. Some work units discovered that their original task plan required change. Just as change to operations in a software class causes no change beyond its interface, changes to responsibilities in process classes caused no turmoil among other work teams. During the project’s execution, unanticipated demands for time arose because collaboration was necessary among teams within the project and with other external projects— the collaboration tax we mentioned earlier. In addition, team members’ geographic distribution added to the level of effort needed for effective communication. Additional effort also went to educating collaborators, troubleshooting interim products, email, using collaborative work-space tools, and holding meetings. The structured walk-through—a miniature, in-progress review—provided an effective technique for achieving collaborator and management understanding. Ultimately, the first increment yielded a successful proof of concept, showing that interoperability among heterogeneous systems was feasible using open, distributed-object computing. As planning for the second increment began, the team leadership found that the detailed process definition from the first increment had substantial reuse value. Upon receiving the Increment 2 assignment, team leaders quickly tailored the Increment 1 process definition for the next task. Thus, the first process definition document became a reusable process reference model, allowing rapid development of level-of-effort and schedule estimates for later increments. Just as process class responsibility statements and product definitions ported easily to a project plan Gantt chart, they readily ported to concise contract language for Increment 2. The significance of these results is that the first increment was successful and fell within cost and schedule tolerances, despite the project’s complexity and the team’s relative inexperience with OO technology. Also significant was the reuse value, measurable in time and money, of using the original process definition document as a process reference model. By the third increment, CIOOT leadership could give the govern-

May/June 2001

ment a cost and schedule estimate within three hours of learning the Increment 3 assignment. CIOOT delivered that increment on time and slightly under budget. We recognize that our findings and our technique have some limitations. Our results come from a small sample—four work increments of one project. At the time we defined our process, the literature on such uses of OO technology was sparse and contradictory. The initial level of effort to develop the process reference model was high, requiring strong faith and leadership before we realized any benefits. Developing the first process diagram took two weeks of intense engagement by 15 team members. In addition, process definition was sometimes contentious, incurring the risk of harming team relationships early in the project. The debates quickly calmed, however, as team members realized they were preventing misunderstandings of potentially greater impact later in the project. Finally, there exist no integrated tools for easily porting and maintaining this technique’s products. Analysis This project supported our fundamental hypothesis about OO technology: Treating logical units of work as classes with responsibilities and collaborations rapidly produced a substantial return on investment. The CIOOT team had neither a formal charter nor the resources to analyze this technique scientifically via metrics and control groups. However, our overall experience was favorable and yielded confidence in our technique’s value to other complex, OO interoperability projects using an iterative, incremental approach. This process definition technique must start with an achievable architectural concept. There must be a vision of what to build before deciding how to build it, as in Grady Booch’s well-known analogy of the doghouse and the skyscraper: don’t buy an industrial crane and hire steel workers until you know that the assignment is a skyscraper. Without clarity of architecture, procedural disputes can never be fully resolved. Benefits Although we didn’t measure this objectively, team members often reported that the

clearly defined process saved them substantial time in executing tasks. For example, because receivers had preliminary knowledge of the nature and format of the senders’ products, they could do early planning and staff training, avoiding time lost in false starts. The implementation of a wellunderstood CM and RM process definition—which even included alerts (as shown in Figure 3)—practically eliminated the uncertainty about when and where products from others would be accessible. For such a complex and team-driven project, technical management was able to use the defined process document and its associated Gantt chart to keep work on track and quickly mitigate risk. In addition, technical management’s familiarization with the process and the principles of the technology helped us resolve issues quickly. Also, management put the work to use quickly. For example, a project plan workbreakdown was ready just a single workerday after we completed the process definition. Generating contract task statements for subsequent increments took a fraction of the normal time because we could reuse large portions of the process definition. Process class leaders could quickly tailor the process diagram for proposed new work, making subsequent task statements much more substantive and accurate. Soon after the second project increment, management acknowledged that the return on investment had been rapid and high. At the time of this writing, CIOOT was ascertaining the portability of the technique to the Government Computer-Based Patient Record (GCPR) project. Project managers concluded that this technique provides a powerful tool to manage borderline conditions between work units, primarily by flushing out assumptions on both sides of a collaboration, making the assumptions explicit, and resolving differences identified. Team status meetings let managers detect departures from the defined process easily and early, so they could nip in the bud wasteful misunderstandings between teams, tangential activities, and dead-end activities. (Departures from the defined process occurred not only in geographically distant groups; collocated teams occasionally started to disconnect as well.)

Additionally, the technique provided contractually clear borderlines between various contractors and subcontractors in terms of their responsibilities and product deliverables. Interfacing As in system engineering, the devil is in the interfaces. Our technique might provide better interface definition if it included association classes (or perhaps collaboration classes) between the work unit process classes. Our technique made collaborations sufficiently explicit in terms of products, but we could improve it by better articulating the collaboration’s responsibility actions— that is, dealing with the collaboration tax. By making communication, education, troubleshooting, and replanning actions explicit in the interface, we could capture level-ofeffort estimates for those associated tasks, further enhancing the accuracy of cost and schedule estimation. As a risk mitigation activity, the CIOOT team originally deployed a staff member from each receiving work unit to its sending work unit for the latter part of the sending task’s work. Conversely, we also sent a staff member from the sending task to the receiving task for the early part of the receiving task’s work. This practice ■

■ ■ ■

Technical management was able to use the defined process document and its associated Gantt chart to keep work on track and quickly mitigate risk.

enhanced the collaborating teams’ understanding of the recipients’ requirements and the senders’ constraints, provided quality assurance in the current mission increment, improved product quality, and resulted in professional growth for staff.

Using the process outside CIOOT The CIOOT team did not have the resources to develop the documentation for formal SEI CMM designation. However, in evaluating the project and its practices, the US Government Accounting Office concluded that the CIOOT process definition technique, when applied iteratively, complied with the spirit of Level 3 in terms of process definition, process training, and continuous process improvement. Our results with CIOOT are relevant to complex projects anticipating iterative or incremental work among diverse or geographically distributed collaborators in a complex May/June 2001

IEEE SOFTWARE

81

Secondary Lessons from the CIOOT Project The principles of object orientation are not as difficult as we imagined. Although our process definition technique does not address the concepts of inheritance or polymorphism, it does effectively introduce OO notions of class, collaboration, interface, information hiding, and responsibilities. ■ Encapsulation of process into classes simplifies complexity. During project execution, this encapsulation eases the shock of process change. Although the classes in this technique articulate processes analogous to operations or methods in a true class, the team thought the metaphor strong enough to call the technique object-based process definition. ■ While discrete process classes aid planning and contract management, day-to-day execution of the process works best with blurred borders. Interim OO technology products are best when the interested parties are actively collaborating. Execution is often a reverberation among process classes rather than a flow through them. ■ Meticulous process definition yields value in near-term cost, schedule, and performance. ■

team environment. In addition, the technique might be useful to project and technical managers somewhat new to the definition of large, complex OO software development projects. Although our results come from experiences with an OO project, the process definition technique itself is potentially applicable to other kinds of projects, developing substantially different products. Limitations The process definition technique we’ve described does have some limitations. One is that there is no single, commercially available tool to support its automation. Keeping the parallel sequences of activities manually synchronized proved a labor-intensive, frustrating chore. The team did attempt sequence diagrams; these were valuable, but so labor intensive that they could not be maintained as the process was modified. The experiences we report here come from one project with four increments; this didn’t provide sufficient data points to allow generalizations. For this reason, we need to monitor our technique’s performance on further work increments—and on a new project. 82

IEEE SOFTWARE

May/June 2001

■

■ ■

■ ■ ■

■

The team started on faith but moved to firm belief in the spirit of the SEI CMM. The first process definition activity calls for continuous commitment and patience. It took 15 leaders—nearly half the team—two intense work-weeks to detail the first process diagram. The activity of process definition can be contentious. For OO projects, it is worthwhile to make OO principles understandable and important to project management. This process definition technique provides a good introduction to those principles. The technique might also be applicable to projects that do not employ the OO paradigm. The collaboration tax is costly and generally not accounted for. In-progress review in the form of structured walk-throughs has near-term value as an educational and situation awareness activity. Such walk-throughs also have longer-term value; they frequently bring to light process improvement opportunities. The technique contributes to continuous quality improvement.

The first time the process is defined for a project, extended, face-to-face workshops led by professional facilitation are a necessity. The attendees must commit to attending the workshop in its entirety, and they must represent each of the identified work units. Such work sessions can be contentious; a strong facilitator who enforces ground rules and mutual respect can be important to success. In the original process definition effort, the team only implicitly accounted for the collaboration tax, treating it as unaccounted responsibilities within classes. However, the CIOOT team soon became painfully aware of the importance of explicitly accounting for this additional time and effort. One possible way to do this would be to borrow the concept of an association class from the UML parlance to explicitly capture the collaboration task between any two work units. According to Martin Fowler,6 an association class enables the addition of attributes, operations, and other features to associations between classes. An association process class between work units could add level-of-effort parameters to capture a collaboration tax estimate. In ad-

dition, briefings, structured walk-throughs, meetings, and troubleshooting would be articulated in association classes.

Acknowledgments This article refines a presentation made originally at the CORBAmed Domain Task Force at the Object Management Group international conference in January 1999. The views expressed in this article are those of the authors and are not official positions of Virginia Polytechnic Institute and State University, Battelle Memorial Institute, or the US Department of Defense.

T

he CIOOT team’s OO-based process definition technique clearly has value beyond workflow diagramming. The initial large investment in effort yields big returns, especially in terms of risk mitigation. The process of defining responsibilities and collaboration products stamps out assumptions and ambiguities that manifest themselves as gaps in the process or duplications of effort. The “Secondary Lessons” sidebar summarizes some of our other findings. We believe this technique is repeatable and portable to similar circumstances, and we encourage others with similar complex project challenges to employ it. The technique will yield the best return on investment in programs or projects that involve multiple, geographically distributed participants working collaboratively in an iterative, incremental fashion. For its next project, the CIOOT team will work on a wide range of consultation services in support of the Government Computer-Based Patient Record program. The government has requested over-arching process definition using the CIOOT technique. This GCPR experience will be more complex than CIOOT, giving the team a chance to test the technique’s scalability. The GCPR program has standardized around the Rational 98i tool suite. The team hopes to work with Rational to explore enhancements and supplements to the Rational Rose tool. With enhancement, we could theoretically use this tool to store process use cases driving process models expressed in UML, collaboration diagrams, and sequence diagrams. For example, an enhancement that reverses inheritance arrows to show workflow would be helpful. This could ease our current labor-intensive maintenance of PowerPoint files. Given success with Rational Rose enhancement, a utility that ports process model information to project-planning applications such as MS Project or Primavera would significantly ease the transfer burden. We encourage feedback from others interested in this technique.

References 1. Capability Maturity Model for Software, Version 1.1, tech. report CMU/SEI-93-TR-24, Software Eng. Inst., Carnegie Mellon Univ., Pittsburgh, 1993. 2. C. Egyhazy et al., “Object-Oriented Analysis and Design: A Methodology for Modeling the Computer-Based Patient Record,” Topics in Health Information Management, vol. 19, no. 1, Aug. 1998, pp. 48–65. 3. K. Keahey, “A Brief Tutorial on CORBA,” www.cs. indiana.edu/hyplan/kksiazek/tuto.html (current 6 Apr. 2001). 4. S.W. Ambler, The Object Primer, Cambridge Univ. Press, New York, 1995. 5. D.S. Newman, “Articles on Class-Based Engineering,” www.technium-inc.com/oo_arts.html (current 6 Apr. 2001). 6. M. Fowler and K. Scott, UML Distilled: Applying the Standard Object Modeling Language, Addison-Wesley, Reading, Mass., 1997.

About the Authors Csaba Egyhazy is an associate professor of computer science at the Virginia Polytechnic

Institute and State University, at the Northern Virginia Graduate Center in Falls Church, Virginia. His main interests are distributed object computing, query processing among heterogeneous databases, and object modeling research. Currently, he is involved with CORBAmed, an OMG special-interest group developing interface specifications for the healthcare domain. Egyhazy is on the editorial board of the International Journal of Computers and Applications. He received his PhD from Case Western Reserve University. Contact him at the Dept. of Computer Science, Virginia Tech, 7054 Haycock Rd., Falls Church, VA 22043; [email protected]. Scott Eyestone is a senior research scientist at Battelle Memorial Institute, Falls Church Office. He is a retired US Air Force officer who served as a clinician, contingency medical support planner, and investigator of emerging information technologies. His interests are in military medical informatics. Eyestone received a Doctor of Optometry degree from Illinois College of Optometry. Contact him at Battelle, 241 Garrisonville Rd., Stafford, VA 22554; [email protected].

Janet Martino manages the Architecture and Standards Implementation Business Area

for the Office of the US Assistant Secretary of Defense for Health Affairs Military Health System Information Technology office. Previously, she was deputy program manager for the Government Computer-Based Patient Record Program—a collaborative interoperability initiative between the US Department of Defense, the Department of Veterans Affairs, and the Indian Health Service. As a medical officer in the US Air Force, Martino has served in Europe as an internal medicine and critical-care consultant, in contingency operations supporting Operation Desert Storm, and in a UN peace-keeping operation in Zagreb, Croatia. She received an MD from the State University of New York at Stony Brook. Contact her at USAF, MC, 5109 Leesburg Pike, Sky 6, Ste. 900, Falls Church, VA 22041.

May/June 2001

IEEE SOFTWARE

83

feature defect detection

Software’s Invisible Users

James A. Whittaker, Florida Institute of Technology

oftware is deterministic. Given a starting state and a fixed series of inputs, software will produce the exact same output every single time those inputs are applied.1 In fact, many of the technologies we apply during development (reviews, formal proofs, testing, and so forth) rely on this determinism—they would not work otherwise.

S Software systems receive input from several kinds of users, only one of which is human. The author discusses other kinds of users, often misunderstood and sometimes forgotten, and the problems that ignoring them can lead to. 84

IEEE SOFTWARE

But if software is deterministic, why do weird things happen? Why is it that we can apply a sequence of inputs and observe a failure and then be unable to reproduce that failure? Why does software work on one machine and fail on another? How is it that you can return from lunch and find your Web browser has crashed when it wasn’t being used? The answer is, of course, that modern software processes an enormous number of inputs and only a small percentage of these inputs originate from human users. When testers can’t reproduce failures, the reason is that they are only resubmitting the human input without sufficient regard to the operating system return codes and input from runtime libraries. When software fails on a new computer after running successfully on another, the reason can only be that the new system is supplying input that differs in content from the old one. And the browser that crashes when you are at lunch

May/June 2001

is responding to input from a nonhuman external resource. Where errors slip in Testers and developers routinely ignore these invisible users or, even worse, do not realize they exist. But it is the mishandling of their inputs that causes hard-to-diagnose periodic system failure and, worse, opportunities for hackers who are all too familiar with such weaknesses in software development practices.2 The inputs from the software environment—the operating system kernel (the part of the OS that supplies memory, file handles, heap space, and so on), runtime libraries, external application programming interfaces (APIs), and the file system—are much the same as inputs from a human user: there are lots of them, many are invalid or produce error codes, and each one should be validated before being processed by application software. The danger in not 0740-7459/01/$10.00 © 2001 IEEE

Typical Reactions to Low-Memory Situations We wrote a tool to simulate low-memory behavior and tested a number of standard industry applications. All the applications failed to gracefully handle unexpected return values from the kernel. Here are some of the results. Scenario 1. We loaded a legitimate page in a Web browser and then denied further attempts to allocate global and local memory. Then we reloaded the same page using the browser’s reload button. The browser returned an “invalid syntax” error message. Diagnosis: the syntax is obviously fine because we reloaded a previously displayed page; however, the developer wrote a global exception handler and the only thing he or she could think of that would cause the exception to be raised was that the user typed an invalid URL. ■ Scenario 2. We selected the Open File dialog from a desktop application’s menu and pointed it to a directory with lots ■

doing so is the same risk we take when we let human users enter unvalidated input: corrupted data, buffer overruns, and invalid computation results.3 To demonstrate input from invisible users, my colleagues at the Center for SE Research and I constructed a software tool that watches Windows programs while they run and identifies events that cross external interfaces. We then executed a number of applications and let the tool log all the activity across their various interfaces. The results are eye opening. For example, Microsoft PowerPoint, a complex and large application for making presentations and slide shows, makes 59 calls to 29 different functions of the Windows kernel (excluding GetTickCount, which was called nearly 700 times) upon invocation. That means a single input from a human user (invoking the application) caused a flurry of undercover communication to be transferred to and from the OS kernel. Certainly, invocation is a special input and requires a great deal of setup and data initialization. But other operations are also demanding on low-level resources: when PowerPoint opens a file, it calls 12 kernel functions a grand total of 73 times (once again excluding GetTickCount, which was called more than 500 times); when PowerPoint changes a font, it calls two kernel functions a total of 10 times. And these are only calls to the OS kernel. PowerPoint also uses a number of other external resources such as dynamically linked libraries in the same manner as the kernel.

of files. The files were listed correctly in the dialog box. Then we blocked access to local and virtual memory and redisplayed the same dialog. The files were not listed (even though they still resided in the same directory). Diagnosis: not enough memory was available to correctly populate the Open File dialog’s display area, but the application had no facility to handle this situation. However, because we did not block access to creating the dialog window, the function completed without listing any files. In scenario 1, the developer realized that the function might fail but did not fully consider all the ways it could fail. Thus, we received an error message that did not describe the real problem and would be useless for diagnosing the failure. In scenario 2, the developer didn’t even consider failure. The function completed as though it were successful, so the user was left wondering what might have happened to the files.

Clearly, the amount of communication between an application and its “invisible” users dwarfs visible human input. Perhaps we could account for many of the mysterious and hard-to-reproduce system anomalies if we treated invisible users the same as we treat human users. If any of those calls produce unexpected return codes, the system in question must handle these events and react appropriately. To test applications’ capabilities of handling unexpected input, we perturbed some of these inputs so that the application in question received legitimate but unexpected return codes and data from its environment. For example, when an application allocated memory, we passed back an unsuccessful return code; when the application opened a file, we told it there were no more file handles; and so forth. Every application we tested this way failed within seconds (see the “Typical Reactions to Low-Memory Situations” sidebar). The diagnosis: many software applications are unaware of the diversity and complexity of their environment. We seem to expect that nothing will go wrong, networks won’t get congested, memory will never run out, and virus writers are just pranksters who can cause no real harm. (See elsewhere for another case study detailing the failure of applications to handle unexpected system call return codes.4) How important is it to handle these situations? Skeptics will argue that inputs usually come in as expected, so that the cost of checking them is not money well spent (for more on checking inputs, see the “ValidatMay/June 2001

IEEE SOFTWARE

85

Validating User Input The most straightforward way to validate user input is to follow the Get Input command with a selection statement (if, case, and so on) that checks validity before continuing. But you must then encapsulate the input routines in a separate module to avoid littering the body of the code with ifs. Remember too that all this checking will slow down your code. If the input is to be immediately stored, it is prudent to hide the storage structure behind a firewall of access routines (that is, make it an object). The access routines are then responsible for not allowing bad data into the structure. A popular but unsafe option is to simply raise an exception, trapping any errors. Beware of the side effects of exception handling, though. Programs tend to lose state information when exceptions occur because the instruction pointer has changed abruptly. When the instruction pointer is restored after the error routine executes, files could still be open and data might or might not be initialized, creating traps for unwary programmers. Preventing the input from ever getting to the application in the first place is possible only at the human user interface. GUIs are specifically designed to do just that. Specialized controls that will only allow integers (or other data types or formats) to be typed are a great way to filter unwanted input without having to write error routines.

ing User Input” sidebar). However, hackers are not so forgiving of our software’s weaknesses. Common attacks such as packet storms that overwhelm memory and stress resources are among a number of ways to cause denial of service by exploiting an application’s lack of attention to its environment. Often, external events such as a specific time (which is a return value of a kernel call and, therefore, an input) trigger logic bombs. How can we possibly test for such things without treating memory and systemcall return values as inputs? This is indeed the bottom line for software developers and testers: you must consider every single input from every external resource to have confidence in your software’s ability to gracefully and safely handle malicious attacks and unanticipated operating environments. Deciding which inputs to trust and which to validate is a constant balancing act. Where do these inputs come from? How should developers and testers handle them? Figure 1 depicts a typical software application and its operating environment. The inner circle represents the application in question. Note that its boundaries are completely contained within the OS. Indeed, modern software communicates with nothing but the OS—all communication with any other external resource must go through it first. Beyond the OS lie four classes of user 86

IEEE SOFTWARE

May/June 2001

(which is the term I will adopt for any external resource with which software can communicate). These are human users, OS users, API users, and file system users. Let’s look at each of these users, the challenges of dealing with them, and the types of inputs they can apply. The human user Human users often do not understand that their input does not actually touch the applications they use. But programmers must understand this or risk inheriting bugs from the programs that really do process human input. Consider keyboard input, for example. First, a keyboard device driver interprets the keystrokes, and then the operating system passes that information through layers of OS libraries and APIs until it finally generates and then passes an event to the target application. The event is all the application sees; the many layers of OS libraries take care of interpreting the inputs from the human and translating them into something the application can deal with. Can things go wrong with this process? Of course—software is involved, after all. Bugs in the OS or code for GUI controls can cause all types of problems for an application expecting only a successful event. For example, what happens when you put too many entries in a list box? The list box will fill up and then either crash, taking your application with it, or generate an error code that gets passed back to your application. If your application is not equipped to handle the error, it will crash—all on its own this time. It is imperative that programmers understand this process and learn which interface controls and APIs are trustworthy and exactly how and when they return error codes when they fail. When developers do not handle GUI control error codes, they are risking denial-ofservice attacks that exploit such holes by finding ways to force the interface error to occur. Even worse, developers might explicitly trust data passed from a GUI control; this opens the door for potential buffer overruns when the user supplies too much data. Do not trust interface controls; their input must be carefully constrained. Compounding this situation are applications that let developers become users by ex-

The API user Similar to the OS user, APIs are external libraries that an application can use to store data and perform various tasks. For example, an application can make SQL queries to an external relational database or use APIs for programming sockets, performing matrix algebra, manipulating strings, and doing any number of other commonly reused functions. These functions usually provide reliable service, but they are also software and can contain bugs that the programmer must consider. Functions can also provide unsuccessful return values that the program must handle properly. For example, a database could be offline or contain corrupted data; a socket connection could drop packets or fail to acknowledge a transmission due to network congestion. Developers must be aware of the possibilities, and testers must figure out how to test them. Trusting other software components leaves our own programs vulnerable to their bugs. Not fully understanding how components behave when they fail subjects our software to unexpected (but legitimate) return codes.

Deciding which inputs to trust and which to validate is a constant balancing act.

The file system user Files (binary or text) are users, and their contents are inputs—so they can also be invalid. Files, even binary ones, are easy to change outside the confines of the applications that create and process them. What happens, for example, when a privileged

Kernel

Application programming interface

rating syste pe Application under test

m

The operating system user As discussed earlier, the OS is the intermediary between all physical users and the application. It also interacts directly with an application by supplying memory, file handles, heap space, and so on. We call this latter part of the OS the kernel. The Windows kernel, for example, exports over 800 different functions, each of which has at least two return values (for success and error conditions). This is indeed a challenge for developers who all too often trust the kernel explicitly. When they allocate memory, they expect memory, not a return code saying “Sorry, it’s all gone.” Every time the program allocates memory, it has to check the return code for success before continuing its task. If it does not, it will behave unexpectedly (or perhaps fail) when low-memory conditions occur. The experiment cited in the “Typical Reactions to Low-Memory Situations” sidebar is good evidence that developers put too much trust in their system calls always behaving as expected. Certainly this trust is often well founded, but when an OS is suffering a malicious attack such as a packet

storm, the application cannot and should not trust them, especially if the application is expected to work safely and securely. But most applications go blissfully unaware of such problems or react by failing.

O

posing functionality that another program can call. In this case, programmers must consider two issues. First, has the calling program supplied valid and meaningful parameters? Obviously, developers should check parameter validity just as they would variable input through a GUI control. Second, are there side effects to calls that preclude other calls from being executed properly? One common shortcoming is making the same call twice in a row. Once a call has opened a file, the software expects that file to be read, not reopened (it was never closed), so a second call causes the software to fail. Because there is no GUI to conveniently protect the application from such invalid inputs, developers must be responsible for it. Human users, whether the typical variety using an application through a GUI or developers using an application through a programmable interface, pose special but familiar problems to software developers and testers. But other types of users are not so familiar, and our handling of them can ensure the robustness of our applications or make them vulnerable.

User interface

Figure 1. The execution environment for application software. All communication must go through the operating system to reach the application.

File system

May/June 2001

IEEE SOFTWARE

87

Data is the lifeblood of software; when it is corrupt, the software is as good as dead.

user changes the permissions on a file that another user is editing? An application’s defenses against corrupted files are usually weak.5 As long as the file extension is correct, the “magic string” is at the top of the file (as an identifier), and the field delimiters are in place, the application often reads the contents without checking them. I cannot overstate the danger of this. Reading files without validating content means that you are introducing unknown data to your software to be used for internal computation and storage. Data is the lifeblood of software; when it is corrupt, the software is as good as dead.

S

oftware applications are controlled by their environment. Unless the system checks the validity of every single input from every single user, the software can fail. But such overzealous defensive programming is probably not feasible for most software vendors. Indeed, based on my experience, many vendors are probably unaware of the

RENEW your Computer Society membership for ✔ 12 issues of Computer

✔ Member discounts on periodicals, conferences, books, proceedings

extent to which their software applications depend on “invisible” users. Still, some precautions are in order: ■

■

During software design, programmers must continually realize when they are going outside the application’s boundaries to get input from users—and not just the human variety. They must understand that, ultimately, all users are software and realize that users can be buggy, so they must not trust their input. They must decide which inputs to verify and which inputs to trust with full knowledge of the consequences. Testers must accept the challenge to simulate as many anomalous inputs as possible that are realizable and pose a threat to the application and the system. This will mean new tools, new techniques, and purposeful consideration of all software users, even the ones we can’t see.

For systems that execute in a networked environment or protect sensitive data, understanding invisible users is as important as understanding human users. Software developers beware: you never know with whom your software is communicating. References 1. R. Linger, H. Mills, and B. Witt, Structured Programming: Theory and Practice, Addison-Wesley, Reading, Mass., 1979. 2. J. Richter, Programming Applications for Microsoft Windows, Microsoft Press, Redmond, Wash., 1997. 3. B. Miller, L. Fredrikson, and B. So, “An Empirical Study of the Reliability of UNIX Utilities,” Comm. ACM, vol. 33, no. 12, Dec. 1990, pp. 32–44. 4. A. Ghosh and M. Schmid, “An Approach to Testing COTS Software for Robustness to Operating System Exceptions and Errors,” Proc. 10th Int’l Symp. Software Reliability Eng., IEEE CS Press, Los Alamitos, Calif., 1999, pp. 166–174. 5. J. Voas and G. McGraw, Software Fault Injection: Inoculating Programs Against Errors, John Wiley & Sons, New York, 1998.

✔ Free membership in Technical Committees

✔ Digital library access at the lowest prices

✔ Free e-mail alias @computer.org

http://www.ieee.org/renewal 88

IEEE SOFTWARE

May/June 2001

About the Author James A. Whittaker is an associate pro-

fessor of computer science at the Florida Institute of Technology, Melbourne. He founded the Center for Software Engineering Research with grants from a number of industry and government funding agencies. The Center performs contract software testing and research focusing on the reasons why software fails and what can be done to make software safer and more reliable. Contact him at the Dept. of Computer Sciences, Florida Tech, 150 W. University Blvd., Melbourne, FL 32901; [email protected].

design E d i t o r : M a r t i n F o w l e r, T h o u g h t Wo r k s

■

[email protected]

Protected Variation: The Importance of Being Closed Craig Larman

T

he Pattern Almanac 2000 (AddisonWesley, 2000) lists around 500 software-related patterns—and given this reading list, the curious developer has no time to program! Of course, there are underlying, simplifying themes and principles to this pattern plethora that developers have long considered and discussed. One example is Larry Constantine’s coupling and cohesion guidelines (see “Structured Design,” IBM Systems J., vol. 13, no. 2, 1974). Yet, these principles must continually resurface to help each new generation of developers and architects cut through the apparent disparity in myriad design ideas and help them see the underlying and unifying forces. One such principle, which Bertrand Meyer describes in Object-Oriented Software Construction (IEEE Press, 1988), is the Open–Closed Principle: Modules should be both open (for extension and adaptation) and closed (to avoid modification that affect clients). OCP is essentially equivalent to the Protected Variation pattern: Identify points of predicted variation and create a stable interface around them. Alistair Cockburn did not know of OCP when he first wrote about PV (see “Prioritizing Forces in Software Design,” Patterns Languages of Program Design, vol. 2, Addison-Wesley, 1996). Furthermore, OCP is what David Parnas really meant by information hiding (see “On the Criteria to Be Used in Decomposing Systems into Modules” Comm. ACM, vol. 12, no. 2, Dec. 1972). 0740-7459/01/$10.00 © 2001 IEEE

OCP and PV formalize and generalize a common and fundamental design principle described in many guises. OCP and PV are two expressions of the same principle— protection against change to the existing code and design at variation and evolution points—with minor differences in emphasis. I am nominating the term protected variation for general use, as it is short and clear. In OCP, the term module includes all discrete software elements, including methods, classes, subsystems, applications, and so forth. Also, the phrase “closed with respect to X” means that clients are not affected if X changes. For example, “The class is closed with respect to instance field definitions.” PV uses the term interface in the broad sense of an access view—not exactly a Java or COM interface, for example. Information hiding is PV, not data encapsulation “On the Criteria To Be Used in Decomposing Systems Into Modules” is a classic that is often cited but seldom read. In it, Parnas introduces information hiding. Many people have misinterpreted the term as meaning data encapsulation, and some books erroneously define the concepts as synonyms. Parnas intended it to mean hide information about the design from other modules, at the points of difficult or likely change. To quote his discussion of information hiding as a guiding design principle: We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each May/June 2001

IEEE SOFTWARE

89

DESIGN

module is then designed to hide such a decision from the others. That is, Parnas’s information hiding is the same principle expressed in PV or OCP—it is not simply data encapsulation, which is but one of many techniques to hide design information. However, the term has been so widely reinterpreted as a synonym for data encapsulation that it is no longer possible to use it in its original sense without misunderstanding it. This article should be called, “The Importance of Information Hiding,” in honor of Parnas’s description of the PV principle. Dijkstra earlier alludes to the principle in the “THE” project, but Parnas gave it focus and shape (Dijkstra, “The Structure of the ‘THE’ Multiprogramming System,” Comm. ACM, 1968). Mechanisms motivated by PV PV is a root principle motivating most of the mechanisms and patterns in programming and design that provide flexibility and protection from variations. Here are some examples. Familiar PV mechanisms PV motivates data encapsulation, interfaces, polymorphism, indirection, and standards. Components such as brokers and virtual machines are complex examples of indirection. Uniform access Languages such as Ada, Eiffel, and C# support a syntactic construct to express both a method and field access in the same way. For example, aCircle.radius might invoke a radius():float method or directly refer to a public field, depending on the definition of the class. You can change public fields to access methods without changing the client code. Data-driven designs Data-driven designs cover a broad family of techniques, including reading codes, values, class file paths, class names, and so forth, from an external source in order to change the behavior of or “parameterize” a system in some way at runtime. Other variants include style sheets, metadata for object-rela90

IEEE SOFTWARE

May/June 2001

tional mapping, property files, reading in window layouts, and much more. The system is protected from the impact of data, metadata, or declarative variations by externalizing the variant, reading the behavior-influencing data in, and reasoning with it. Service lookup Service lookup includes techniques such as using naming services (for example, JNDI) or traders to obtain a service (such as Jini). This approach uses the stable interface of the lookup service to protect clients from variations in the location of services. It is a special case of data-driven designs. Interpreter-driven designs Interpreter-driven designs include rule interpreters that execute rules read from an external source, script or language interpreters that read and run programs, virtual machines, neural network engines that execute nets, constraint logic engines that read and reason with constraint sets, and so forth. This approach lets you change or parameterize a system’s behavior through external logic expressions. The system is protected from the impact of logic variations by externalizing the logic, reading it in (for example, rules or a neural net), and using an interpreter. Reflective or metalevel designs An example of a reflective or metalevel design includes using the java. beans.Introspector to obtain a BeanInfo object, asking for the getter Method object for bean property X (that is, the method getX), and calling Method.invoke. Reflective algorithms that use introspection and metalan-

You must pick your battles in design, be they at the macro-architectural level or the humble instance field.

guage services protect the system from the impact of logic or external code variations. We could also consider this a special case of data-driven designs. Pick your battles As an example of PV’s application, a client explained that the logistical support application used by an airline was a maintenance headache. There was frequent modification of the business logic to support the logistics. How do you protect the system from variations at this point? From the mechanisms to support PV (data encapsulation, interfaces, indirection, …), a rule-based design was chosen: A rules engine was added to the system, and an external rule editor let the subject matter experts update the rules without requiring changes to the source code of the system. Low coupling and protection against variations is not motivated in all areas. You must pick your battles in design, be they at the macro-architectural level or the humble instance field. A good designer can identify the likely points of instability or variation and apply PV to those points but not others. Otherwise, effort is wasted and complexity may arise (and with it, the chance for defects). For example, I recall being surprised by the occasional use of static public final fields in the Java technology libraries (after spending many years with the Smalltalk libraries). Some might be poorly conceived, but some, such as the Color static fields red, black, white, and so forth, are extremely stable; the likelihood of instability is so low that making them private and adding accessing methods is just object purism. As a counterexample, I know of a pager-message-handling system in which the architect added a fancy scripting language and interpreter to support some flexibility. However, during rework in an incremental release, the complex (and inefficient) scripting was removed—it wasn’t needed. Judicious PV and the Diamond Sutra Constantine’s guideline to design with low coupling is a truly core prin-

DESIGN

ciple of design, and it can be argued that PV derives from it. We can prioritize our goals and strategies as follows: 1. We wish to save time and money, reduce the introduction of new defects, and reduce the pain and suffering inflicted on overworked developers. 2. To achieve this, we design to minimize the impact of change. 3. To minimize change impact, we design with the goal of low coupling. 4. To design for low coupling, we design for PVs. Low coupling and PV are just one set of mechanisms to achieve the goals of saving time, money, and so forth. Sometimes, the cost of speculative future proofing to achieve these goals outweighs the cost incurred by a simple, highly coupled “brittle” design that is reworked as necessary in response to true change pressures. That is, the cost of engineering protection at evolution points can be higher than reworking a simple design. My point is not to advocate rework

and brittle designs. If the need for flexibility and PV is immediately applicable, then applying PV is justified. However, if you’re using PV for speculative future proofing or reuse, then deciding which strategy to use is not as clear-cut. Novice developers tend toward brittle designs, and intermediates tend toward overly fancy and flexible generalized ones (in ways that never get used). Experts choose with insight—perhaps choosing a simple and brittle design whose cost of change is balanced against its likelihood. The journey is analogous to the well-known stanza from the Diamond Sutra: Before practicing Zen, mountains were mountains and rivers were rivers. While practicing Zen, mountains are no longer mountains and rivers are no longer rivers. After realization, mountains are mountains and rivers are rivers again.

PV is a fundamental design principle that applies to everything from the largest architectural concerns to the smallest coding decision. Furthermore, it underlies the motivation and advice of most other patterns and principles. As Parnas explained 30 years ago— and as has resurfaced in the writings of Meyer and Cockburn—each generation of software developers needs help seeing mountains as mountains again —especially after four years of computer science and 500 patterns!

Acknowledgments OCP, as described here, was brought to my attention by Bob Martin in The Open-Closed Principle: C++ Report, SIGS Publications, 1996.

Craig Larman is director of process and methodology at

Valtech, an international consulting group. He holds a BSc and an MSc in computer science, with research emphasis in artificial intelligence. He is the author of Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design, and he is writing a second edition that will include OCP/PV as one of the fundamental design principles. He is a member of the IEEE and ACM. Contact him at [email protected].

How to Reach Us Writers For detailed information on submitting articles, write for our Editorial Guidelines (software@ computer.org), or access computer.org/software/author.htm. Letters to the Editor Send letters to Letters Editor IEEE Software 10662 Los Vaqueros Circle Los Alamitos, CA 90720 [email protected] Please provide an email address or daytime phone number with your letter. On the Web Access computer.org/software for information about IEEE Software.

Subscription Change of Address Send change-of-address requests for magazine subscriptions to [email protected]. Be sure to specify IEEE Software. Membership Change of Address Send change-of-address requests for the membership directory to [email protected]. Missing or Damaged Copies If you are missing an issue or you received a damaged copy, contact [email protected]. Reprints of Articles For price information or to order reprints, send email to [email protected] or fax +1 714 821 4010. Reprint Permission To obtain permission to reprint an article, contact William Hagen, IEEE Copyrights and Trademarks Manager, at [email protected].

May/June 2001

IEEE SOFTWARE

91

country report Editor: Deependra Moitra

■

L u c e n t Te c h n o l o g i e s

■

d . m o i t r a @ c o m p u t e r. o r g

China’s Budding Software Industry Dehua Ju

C

hina’s booming economy is spurring the country’s demand for information technology. Almost 6.5 million personal computers were sold and 8.6 million units produced in China in 2000, continuing an annual increase of more than 25 percent since 1993. Economists expect China to become one of the biggest PC markets in the world as it enters the new century, which is a unique opportunity for growth in both enterprise and personal computing. Our software market has grown at an average rate of 35 percent over the last four years, with no indication of slowing down. Total sales approached US$6.75 billion last year and $9 billion is expected this year (sales of brand-name software packages were $2.12 billion). Since 1997, the home PC has gained unprecedented popularity here, occupying nearly 35.2 percent of the total PC market in the last year. Internet applications have become a driving force in home PC purchases. Currently, China has 22.5 million Internet users and 80 million cable TV users, averaging 21 email messages received and 10 sent per user, per week. However, the lack of application services presents a sharp contrast that opens a great potential market for the local software industry. China’s competitive advantages Although the total output value of most Chinese software enterprises is currently much lower than that of their counterparts 92

IEEE SOFTWARE

May/ June 2001

in developed countries, our companies have an exploitable asset: people. An enormous body (470,000) of well-educated software professionals can be steadily replenished and expanded from the 37,000 computer science bachelors, masters, and doctoral graduates each year. Our other competitive advantage comes from its cheap labor costs. A recent survey reported that Chinese software workers earn between $30,000 and $35,000 per year and $20 per hour—significantly less than equally qualified Indians, who earn $65,200 per year and $36 per hour. Although price is not the sole factor for business competition, sometimes it makes a difference. Some famous Indian software giants are even considering establishing their software centers and think tanks here to leverage their favorable position in the global market. The people dimension The IT field is highly desirable to young Chinese workers owing to its high income potential. In Chinese universities and in many continuing-education schools, computer science is a popular major. To encourage this trend, China has a national examination system for certifying programming skills. This means that we have a vast reservoir of human resources in the computer application fields. Organized properly and combined with effective management, up-todate technology, productive environments, and good market channels, these resources could greatly affect software development. Confronted with a global shortage of IT workers, our young software industry might 0740-7459/01/$10.00 © 2001 IEEE

COUNTRY REPORT

even be able to contribute to offshore software development. The competition for talent in the Chinese software community is as serious as it is in India. We can attribute this to the quick increase in numbers of computer applications and university graduates, which still lag far behind the demands from industry—and the dotcom boom in recent years is further aggravating that gap. Network companies don’t balk at paying high prices to attract the talent they require, which creates expectations for large salaries and a high turnover rate (a 10- to 20-percent attrition rate is the norm in most software companies). It became a big challenge for most companies to find qualified software engineers even at abnormally high pay rates. As a remedial measure, many companies are considering building up their own training centers or programs. To keep core knowledge, some companies have provided stock options to keep key employees. In the new century, China plans to carry out its so-called “talent strategy,” which is based on future economic competition. Highly qualified human resource development is now a national strategic task, and an important measure of it is to industrialize IT education and training here, as India has done. To nip this problem in the bud, China needs an industryoriented education infrastructure to compensate for gaps in university education. Toward that end, some famous Indian IT education and training companies have moved here in cooperation with famous Chinese universities. Under the talent strategy, software productivity and quality will be a national priority, with special stress put on such skill upgrading as process improvement, project management, quality assurance, and system analysis and design. Focusing on quality and process maturity To differentiate themselves from other companies in a competitive market, the recent strategy of many software companies is to improve their

business images, especially in such areas as quality management and brand fame. Most mainstream enterprises received ISO 9000 certification in the last two years. The next target is CMMbased capability maturity evaluation. Right now, CMM and project management training are the hottest topics in China’s software community. Only a few companies have passed CMM evaluation—some have engaged in CMM-based software process improvement, and some foreign companies have started their CMM consulting and training businesses here. Most software companies want to be viewed as learning organizations that proficiently apply the best SE practices to their software development. Software certification, ISO 9000, CMM, and SPICE are natural targets for most software companies with business ambitions. The government promises subsidies to software enterprises that engage in CMMbased capability maturity evaluation. To promote software process improvement, the Shanghai Software Quality Consortium was established last January to join the international software engineering family. Also, the first SPIN organization and Quality Assurance Institute Federation Chapter in China are also being built in Shanghai. A series of training and professional development plans are under consideration to encourage SE professionals to master the state of the art and earn world-recognized certifications such as the QAI’s Certified Quality Analyst and the IEEE Computer Society’s CSEP. The Project Management Institute has established three chapters, in Beijing,

The competition for talent in the Chinese software community is as serious as it is in India.

Shanghai, and Guangzhou. Hundreds of professionals took part in the two PMP certification exams that PMI offered in China last year. The current scenario: Linux and other trends China established several Linuxrelated organizations and users groups last year, including the China CoCreating Software Federation (sponsored by the government-funded State 863 Project). Its main members include Linux vendors such as China Software Company, Red-Flag/CAS, TurboLinux, and BluePoint. Linuxrelated projects are listed as a specially subsidized item in the government funding plan, at both central and local levels, stressing the independent copyright owner. Chinese Linux versions and Web sites have sprung up like mushrooms after rain. Almost all famous international Linux organizations have opened their Web sites, and some have built development centers here. In four months at the end of last year, TurboLinux sold 20,000 copies through federal software stores, ahead of Windows 98 and NT. Chinese Linux users roughly exceed two million, and roughly 10 percent of new PCs in China will have Linux preinstalled this year. Linux’s penetration here is impossible to assess accurately because it is, of course, copied from machine to machine. The China Software Industry Association will sponsor the first appraisal of shareware and freeware here later this year. The main reasons behind the Linux trend are security and nationalistic concerns, as well as to develop our own operating systems and break Western domination and monopoly—especially from Microsoft. Last year a rumor spread through China: the Chinese government was contemplating a ban on Microsoft’s Windows 2000 in all government ministries and adopting Linux as an official operating system. In IT industry circles, free and open source Linux is considered the most promising candidate to replace Windows as China’s operating system. It is also the most economical way to May/June 2001

IEEE SOFTWARE

93

COUNTRY REPORT

introduce advanced, Internet-capable computer systems to third-world countries that cannot afford proprietary software. It also saves us from a vendor lock-in with Microsoft. Moreover, our software industry can add its own value to Linux and play

tion dissemination in the Internet age. The so-called C2C (Copy-to-China, in which Chinese businessmen quickly follow Western business models) mode became a shortcut to success for most IT companies. In China, as in the US, e-commerce, dotcom, B2C,

Information Sources (Mostly in Chinese) China Software Industry Association: www.csia.org.cn Information Industry Ministry: www.mii.gov.cn China Economic Information Network (members only): www.cei.gov.cn China Internet Network Information Center: www.cnnic.net.cn Center of Computer & Microelectronic Industry Development: www.ccidnet.com.cn China Enterprise Information: www.chinabbc.com.cn Software Offshore Business Union of Shanghai: www.sobus.com.cn

a significant role in the Linux world. Last year, the Chinese Academy of Science’s Institute of Software unveiled the world’s first 64-bit Chinese Linux operating system. The release of this native server software, Chinese Penguin64, represents a major advance in Linux development and a significant milestone in China’s efforts to build an indigenous software industry, combining the advantage of open source computing and unlimited 64-bit processing power with native Chinese language capability. Penguin64 is set to become the new computing standard for business and government, so that China can accelerate the pace of local software development and further its ambition to be a software producer for the world market. Motorola and TurboLinux are jointly developing the first Chinese-language version of an embedded Linux operating system to run on the PowerPC 8240 microprocessor. The Linux-based platform will target Internet applications such as interactive TV, global positioning satellite receivers, cellular phones, and other networking equipment, along with the emerging information appliance market. The increase in networking companies is a noticeable phenomenon here. Currently, there are 300 Internet service providers and more than 1,000 Internet content providers. This is mostly attributable to rapid informa94

IEEE SOFTWARE

May/ June 2001

B2B, click and mortar, CRM (customer relationship management), and SCM (supply chain management) have become buzzwords for entrepreneurs. Network stocks were hotly pursued in the stock market for a time. In view of its potential market, the next big thing will be wireless Internet access—we already have 85 million cellular phone users. As in Western countries, a big dip will hit our networking companies this year. Internet businesses will consolidate through bankruptcies, mergers, and acquisitions. Some software and network companies are considering a strategic shift toward becoming application service providers and seeking a path-to-profit mode. This will help enterprises speed up the pace toward e-business. In line with global trends, the Chinese government has placed special emphasis on boosting the IT industry. “Modernization driven by informatization” has become a national guiding principle in economic construction. A recent topic is the country’s ambitious plan to turn its software industry into a new “pillar” industry in the 21st century. Through wellfunded federal “golden projects” and city-sponsored “information port or digital city plans,” the information infrastructure is being put into place. Shanghai is planning to send xDSLbased wideband access to four million households. Such billion-dollar proj-

ects involve all sorts of application domains: finance systems, tax collection, social insurance, business and trade automation, public information services, and remote education, among others. Last year, the government initiated the “government on Internet” and the e-business-centered “enterprise informatization and enterprise on Internet” project. There are now more than 265,000 Web sites established in China, among them 1,100 sites for e-business and 2,000 sites for government. Problems and roadblocks Obviously, chance and challenge coexist here, and many factors could restrain the pace of progress. The main roadblocks to a robust software industry include the weakness of our overall strength in participating in global competition, insufficient investment, rampant piracy, and lack of strong policy support and promotion from the government. In 1999, China exported only $130 million in software, a big lag behind similar countries such as India. According to a survey in the People’s Daily Newspaper, 26 percent of Chinese software companies believed illicit copying, sharing, and unauthorized installation of software is the most significant barrier to their development. Big losses in revenue caused by piracy have seriously constrained further R&D investment from local companies into software products. Also, 19.6 percent of companies complained that a lack of money impeded their growth. Startup companies thirst for venture capital, and small businesses yearn for unblocked channels for bank loans. Another crucial factor is the West’s dominance of our current software market, particularly in system software. Imported brands occupied almost 67.6 percent of the software product market here. This situation will persist until we achieve technological parity with the West. A few Chinese-made software packages have already been successful—for example, Founder’s Chinese Typesetting System, Sun Tendy’s Chinese Star, Stone Rich-

COUNTRY REPORT

sight’s Richwin, and some native financial and accounting packages. The top three domestic software companies are UFSoft Group, Founder, and NEU Alpine, whose revenues were $56.0, $49.4, and $36.8 million, respectively. These companies’ experiences show that innovation, market analysis, venture capital, R&D efforts, and compatibility with local application practices are all important conditions for success. Domestic software companies have their superiority and will continuously survive in the local application service market. The promising application domains include banking systems, IC card applications, financial packages, e-business, CAD/CAM, communication software, and multimedia applications. The small scale and size of most Chinese software companies is another concern. Currently, there are about 5,000 software companies and nearly 470,000 professionals located mainly in our big cities (often in high-tech software parks). Sixty percent are private enterprises, 10 percent are foreign vendors, 55 percent are small start-up businesses with fewer than 50 employees and short histories, and 42 percent have between 50 and 200 employees. They often fail to submit winning bids on large contracts with critical requirements because of lack of experience in systems integration and process management. They urgently need more upto-date information, advanced technologies, high-level software engineers, and integrated solutions to improve their efficiency and competitiveness. Currently, the average yield of a software developer is about $25,000 per year, roughly equal to 1/7 to 1/8 of the outcome of a Western counterpart. Although this could be partially attributed to low labor costs, inadequate organization and management are other important reasons. Enabling initiatives Faced with great business opportunities and their accompanying obstacles, China’s software industry should adopt active measures to keep up the pace with the current world trend. To regain initiative, the government as an

industry promoter is working out new development plans and adopting a series of prop-up measures. Last summer, the State Council announced a new preferential policy that fosters our software and IC industries, including preemption of IPO status, venture capital support, tax exemptions and reductions, software export promotion, guidelines for technology development, a talented person system, bonus and stock options for core employees, qualification of software enterprises, and protection of intellectual property rights. Foreign companies will enjoy equal treatment. Since then, some local governments have published their own promotion policies as a further supplement—for example, Document No. 54 from the Shanghai Municipal Government exempts from taxation personal income and rewards for key software engineers. To unite our scattered strength and integrate our advantageous resources, the government has established 19 software parks throughout the country. Comprehensive and meticulous planning, special policies, and a whole range of supporting services plus billions of dollars invested in infrastructure construction offer a perfect industrial environment for software development. The near-term objectives are $21 billion in annual software sales, more than $1 billion of software exported per year, 10 software companies cultivated that have an annual output value up to $120 million, more than 50 famous software brands, and some key homemade software packages for mission-critical systems.

Toward a more mature industry By recognizing its own disadvantage, our software community is adopting active and realistic measures toward a more mature industry. To open up the global outsourcing market, it knows that there are two main impediments: market channels and improved process and quality management. To join independent actions into a concerted effort, the Software Offshore Business Union of Shanghai was established in the Shanghai Putong Software Park to converge 47 software companies and 4,500 software engineers under one banner. It has set up a resident office in California’s Silicon Valley and started to develop a win–win relationship with foreign business partners and brokers. The government provides preferential policy to encourage students to return from abroad and foreign investors to start up their businesses here. The union will bring in state-ofthe-art technology, modern management ideas, and many business opportunities. To develop our local software industry, we shouldn’t return to a closed-door policy. As a part of the information age, it should instead cooperate with other countries to attract more overseas investors, learn from their experiences, and absorb whatever technologies are needed to develop its own software industry. This includes developing and consolidating business partnerships with the outside world.

Dehua Ju is vice president and general manager of ASTI

To develop our local software industry, we shouldn’t return to a closed-door policy.

Shanghai. He is also a professor of computer science and business at the East China University of Science and Technology and a visiting research scientist and supervisor of CS doctoral students in the Institute of Software at the Chinese Academy of Science. His current interests include CASE tools, advanced software tools and software engineering environments, software automation, network computing, multimedia applications, and e-business. He received his diploma in radio electronics and did postgraduate work in electronic physics at Tsinghua University. He is a member of IEEE Software’s Industry Advisory Board, the IEEE Computer Society, and the ACM. Contact him at ASTI Shanghai, 5F, Bldg.10, 471 GuiPing Rd., CaoHeJing Hi-Tech Park, Shanghai 200233, P.R.China; [email protected]; www.asti-global.com. May/June 2001

IEEE SOFTWARE

95