June)

manager Editor: Donald J. Reifer

■

Reifer Consultants

■

[email protected]

A Little Bit of Knowledge Is a Dangerous Thing Donald J. Reifer

A

fter studying knowledge management in recent years, wondering if it could help me manage software projects better, I’ve concluded that indeed it can. To illustrate this point, I’ve used Table 1 to list potential knowledge management benefits and drawbacks organized around the classical functions software managers perform. Most articles on this topic focus on strategic, not tactical goals, describing potential benefits at the 40,000-foot level. But to determine where you and I as software managers can use this technology to deliver products on time and within budget, the focus must be closer to the 100-foot level. You’ll note that the table lists a “facilitating” function. That’s because project managers need to bring in the right people at the right times to get the job done. Facilitation in this sense is high on any good manager’s priority list. For example, as managers, you and I would need to coordinate with our legal departments to have attorneys available to review each subcontract. Now, let’s see how knowledge management can make your life easier, too. Knowledge management—hype or reality? After I constructed this table, I was skeptical: with these drawbacks, do managers ever take advantage of this technology? So, I talked to members of my peer network, mostly software managers with profit-and14

IEEE SOFTWARE

May/June 2002

loss responsibility in big organizations. It’s a good thing I stayed open minded; most people I contacted in organizations that had mounted knowledge management initiatives were ecstatic about the results. Here are several bits of what I heard: ■

■

■

“We made a project knowledge repository available to all players on our intranet. Once we organized it into collections, useful information was available immediately—right when we needed it. Collaboration was fast and easy. Setting up and maintaining the collections using taxonomies that the corporation created was relatively painless.” “We had a lot of data that we needed to keep track of from old projects. Using existing knowledge management tools, we could mine the data quickly by discovering patterns and relationships that weren’t immediately apparent.” “My firm created a knowledge base of information highlighting critical success factors for project managers. When I first viewed them, I said ‘so what?’ But when I played with the system, I found it intuitive and useful. I could take messy data from my project and make sense of it once I played it through the system. I could also reason with the system based on heuristics derived from relationships.”

Not all the comments were this good. I also heard the following: ■

“Here we go again. Upper management 0740-7459/02/$17.00 © 2002 IEEE

MANAGER

■

must have taken a seminar. Another initiative— more time wasted at the worker level.” “While it sounds good, I am too busy to look at knowledge management. I’ll play the wait-and-see game and hope it blows over soon.”

that a little bit of knowledge about knowledge management can be dangerous. I’m not too bashful to seek help from those who have tackled transfer of this technology, building on their shoulders rather than repeating their mistakes.

Y

You can imagine some of the rest. Give a manager a chance to purge with a peer and that’s exactly what happens. But, the comments made me think twice about how I could use this technology to improve my ability to deliver. My research into the use of knowledge management had panned out. Instead of joining the naysayers, now I can think of a dozen ways the technology can improve my performance. Making it work I’m also convinced that I’ll need help. Not well versed in knowledge management, my staff had no experience with the tools and techniques. My contacts at large companies were lucky. Their corporate initiatives brought in experienced people who

developed the processes project managers need to structure the knowledge base of information serving as their initiative’s foundation. These experts also developed the training and incentives needed to put the technology into practice. I am also skilled enough to know

ou won’t get me to say that knowledge management is a cure-all. But I will recommend that you consider using it—especially if your firm has invested in it. All too often, I’ve seen project after project get into trouble because of miscommunication. People are just too busy to tell each other what they’re doing. Using knowledge management, you can make this information readily available, especially if your firm helps you put the systems and procedures into place to overcome the challenges associated with technology transfer. Keep an open mind, and check out this technology. You might like it. Don Reifer is president of Reifer Consultants and visiting associate of the Center for Software Engineering at the University of Southern California. Contact him at [email protected].

Table 1 Project manager’s assessment of knowledge management technology Management functions Project planning

Organizing Staffing Directing

Controlling Facilitating

Potential upsides Captures organizational knowledge and makes lessons that are often learned the hard way available for use in an organized and systematic manner. Determines patterns of organization that work best based on domain knowledge. Makes it easy to identify people with the skills and knowledge required to fill open slots in the organizational structure. Makes it easy to deduce leadership and teamwork abilities. Identifies and streamlines optimal communication patterns in the organization. Exploits lessons learned relative to past experience. Creates benchmarks to compare performance against, both corporate and competitive. Identifies linkages between work processes and highlights techniques that improve efficiency and effectiveness.

Possible limitations Hard to capture lessons learned. Few incentives to use knowledge, especially when under pressure to get plans finalized. Your customer might require an organization that doesn’t adhere to preferred patterns. An individual’s ability to fit into a team is something that only the people he or she will work with can fully evaluate. Leadership and teamwork are qualities that need to be developed, not inferred. While communications can be streamlined, breakdowns often occur due to people issues that must be anticipated to address. Benchmarks might be misleading because they might be based on norms instead of specifics. Good project managers can facilitate problem solving because they anticipate problems and bring in experts to work on them.

May/June 2002

IEEE SOFTWARE

15

requirements Editor: Suzanne Robertson

■

The Atlantic Systems Guild

■

[email protected]

Requirements and Psychology Chris Rupp “Well, I thought you meant...” “No, that’s not what I said...” sounds familiar to any requirements engineer. Chris Rupp decided to borrow some ideas from the discipline of psychotherapy to help pin down what people really mean. Here’s a good example of how we can use well-developed ideas from other disciplines to help improve our own. —Suzanne Robertson

F

ulfilling your customers’ interests determines your market success, but how do you find these requirements effectively and efficiently? As simple as this question sounds, answering it in daily practice is difficult. Often, stakeholders are interviewed about their requirements or asked to write them down, but this approach rarely uncovers the real requirements that reflect a customer’s true interests or needs. We need a way of getting information about the customers’ core desires—conscious, unconscious, even subconscious. The hottest sellers are products that fulfill these desires.

Linguistics Communication between people is never easy, especially when it’s about requirements for complex systems or products. To help overcome communication problems, I recommend a method from the area of neurolinguistic programming (NLP), which belongs to the field of psychotherapy. (For further research into this topic, please see R. Bandler and J. Grinder, The Structure of Magic, Science and Behavior Books, 1990.) This conversational method tries to decode what a person means when speaking. According to NLP, mistakes in statements can happen in two places: 16

IEEE SOFTWARE

May/June 2002

■ ■

perception of facts (reality) linguistic representation of one of these perceptions (personal reality)

The goal of semantic language analysis is to uncover formulations borne of subjective experience and replace them with clear, objective formulations. Since 1995, our company, Sophist GmbH, has applied language models such as the Requirements Engineering (RE)-Metamodel to areas of computer science. The result has been simple, easy-to-use rules for prose requirements in project development. Using these adaptable rules, we can systematically review requirements formulated in natural language so that errors or ambiguities never see the light of day. The RE-Metamodel Figure 1 shows the RE-Metamodel, summarizing the most frequently observed language phenomena linguistically. Each of these phenomena indicates linguistic defects. I will explain an example of each type of defect and how you can recognize them in your own work. Deletion We all use the process of deletion to reduce the world to dimensions that we can handle. In requirements engineering, however, we must know exactly what information we’ve lost, or we run the risk of a mis0740-7459/02/$17.00 © 2002 IEEE

REQUIREMENTS

Deletion ➯ Presuppositions ➯ Incomplete comparatives and superlatives ➯ Modal operators of possibility ➯ Modal operators of necessity ➯ Incompletely defined process words

Generalization ➯ Universal quantifiers ➯ Incompletely specified conditions ➯ Nouns without referential indices

Distortion ➯ Nominalizations

Figure 1. The RE-Metamodel, summarizing the most frequently observed linguistic defects.

understanding. One way to find deleted information is to examine a sentence’s process words, or verbs. To be complete, a process word usually requires an argument or a noun phrase. Consider the following: “The system shall report data loss.” The process word “report” is completely defined only if the following questions are answered: Who reports what to whom? What is reported? When? How? For each requirement, you must be sure that the process word in it is a satisfactory and clear-without-ambiguity definition of the actual process. Identifying and roughly specifying the processes that your system should carry out will guide you in asking questions that clarify meaning. If dynamics and complex processing rules play a deciding role in your system, pay special attention to process words. They can lead you to a clear definition of when and in what condition your system performs its processes. Generalizations In the generalization process, requirements seem to apply to the system as a whole, but in fact they usually apply to just a small piece of it. Typical indicators of generalization are universal quantifiers: parts of statements broadly applied to all of an occurrence’s incidences. Representatives of linguistic quantifiers include concepts such as “never,” “always,” “no,” “every,” and “all.” The danger in using them is that the specified behavior does not always apply to all the referenced objects in a group or set. Consider the following: “Each signal shall be labeled with a time stamp.” On the

basis of the keyword “each,” a question immediately arises: Are there one or more special cases in which the time stamp is not required? You probably won’t find many universal quantifiers when searching your text, which is not necessarily a good sign. Inspect sentences that don’t contain explicit statements about the number of objects for which the specified reaction occurs. They often contain an implicit assumption that the specified reaction is valid for all relevant objects. Sometimes knowing how much time and effort to spend on special cases and exceptions is hard. It all comes back to assessing the risk of not asking the questions. In cases where safety of life and limb is involved, you can’t afford to miss exceptions. Distortion The problem of distortion appears almost exclusively in the form of nominalization, which occurs when a process is reformulated into an event. A nominalization can change a statement’s meaning and cause important information regarding a process to be lost. Linguistically, nominalization is a process word (verb or predicate) molded into an event word (noun or argument). Consider the following: “After a system breakdown, a restart shall automatically be initiated.” The processes behind the nouns “system,” “breakdown,” and “restart” actually consist of a system breaking down and a system restarting. But, how is the restart being performed? Who initializes it? What ends it? Using nominalized terms for a complex process is fine—if the process

is clearly defined. This definition should not leave any margin for interpretation of the process and should clarify its progression as well as all input and output parameters. The goal is not necessarily to avoid nominalizations but to use them only if the process lying behind them is clear. Nominalizations often appear in domains with an extensive technical language, so check your field’s terminology. Certainly you will identify most specialist terms as nominalizations that hide a large amount of specialized knowledge. Applying the metamodel Many companies have successfully used the RE-Metamodel in various project areas: Deutsche Flugsicherung, Swisscontrol, and Eurocontrol (air traffic control systems); Deutsche Post (logistic systems); Kreditwerk and Bausparkasse Schwäbisch Hall (societal systems); and Stoll GmbH (knitting machines). In all these different areas, the REMetamodel helped systematize the engineering of stakeholder needs. Stakeholders used additional templates to formulate requirements, which in turn clearly defined how to finalize them (see www.sophist.de for more information). Clear preconditions on how to elicit and accurately document requirements is especially helpful in cases where they are not noted in the stakeholders’ primary language. Most analysts and stakeholders love to receive precise working instructions because specifying a new system can be a complex and nonintuitive process. You can apply the RE-Metamodel in three ways: May/June 2002

IEEE SOFTWARE

17

REQUIREMENTS ■

■

■

Dialogue. The analyst immediately checks each stakeholder statement for linguistic effects and studies missing facts that appear to be important. Thus, interviews quickly get down to the nitty gritty, efficiently eliciting the required knowledge and promptly revealing knowledge gaps. This technique requires an experienced analyst. The interviewee should not notice that the analyst is systematically inspecting all statements for linguistic effects. To do this, the analyst must be experienced enough to identify and ask essential questions. Inspection of written requirements. Using linguistic rules, the analyst inspects already existing requirements for omissions, lack of clarity, and so forth according to the metamodel’s rules. Combination. Of course, you can combine both approaches.

Ultimately, the analyst decides how to use the RE-Metamodel, adapting it to each particular situation on a case-by-case basis. With some concessions, the RE-Metamodel can help represent a complete, faultless set of requirements using the informal notation of natural language. Thus, the door is opened for the analyst and stakeholder to work together to review and criticize the problem’s working description. Usually, the NLP approach is imparted at a two- or three-day training course. Afterward, you can successfully apply it immediately because you don’t have to learn a new way of thinking, you just have to explore your knowledge of language. You’re learning a model to make that knowledge more accessible and some refinements to help you apply it to RE. For the first application of the method after training, we recommend using it to analyze already existing documents. On the basis of existing requirements, you can test each statement for linguistic effects and scrutinize it before interviewing stakeholders. A few days of practice will integrate the RE-Metamodel into 18

IEEE SOFTWARE

May/June 2002

your thinking enough for you to examine and scrutinize your customers’ statements in real time. What makes a good analyst? Knowing that perfect communications are nearly impossible relieves you of trying to achieve a perfection you never could reach. Managing the incompleteness of communication is core to mastering the RE process. The point of developing approaches such as the RE-Metamodel is to help systems analysts be more effective in doing a complex job. Good analysts mediate between worlds. They investigate the stakeholders’ needs while discovering, formulating, and inventing requirements. They then translate and transmit this information to the world of system development and informatics. Analysts do much more than translate stakeholders’ wishes into requirements, however. Excellent communicative and therapeutic skills are expected. In cases of diverging opinions between stakeholders, analysts must work out a common essence and develop acceptable solutions for all people involved. They must overcome the stakeholders’ hindrances and fears; every new system carries the potential to create fear by scrutinizing previous positions and habits. Analysts should have experience in group dynamics, conflict management, and team management apart from sound IT and domain know-how.

Analysts do much more than translate stakeholders’ wishes into requirements, however. Excellent communicative and therapeutic skills are expected.

Good systems analysts have a complete suitcase of elicitation techniques from various sectors. This suitcase should contain therapeuticsector approaches (NLP, family therapy) as well as creativity techniques (brainstorming), behavioral science methods (observation techniques), and computer science approaches (use-case-driven modeling, prototyping techniques). A good analyst’s working method is never really predictable. He or she starts the conversation with a stakeholder and decides, based on the situation, which technique to use. Factors influencing that decision include ■ ■

■

■

Power and trust relationships within the team Whether the demanded information is lodged in the conscious, unconscious, or subconscious Whether the stakeholder is good at explaining knowledge (prefers to work with concrete examples or abstract concepts) Whether the system is concerned with dynamic–active procedures or static information

The analyst should develop a flair for using the right technique in the right situation. A mixture of many different techniques can bring revolutionary success (see www.systemsguild.com).

T

o be a successful requirements engineer, be prepared to always be in a learning situation, accept change, and never do things by rote. The variety of techniques, stakeholders, and domains make the job of RE an increasingly challenging and enjoyable one.

Chris Rupp is the managing director of Sophist GmbH and

Sophist Technologies GmbH. Contact her at Sophist Gesellschaft für innovative Softwareentwicklung mbH, Vordere Cramergasse 11-13, 90478 Nuremberg, Germany; [email protected]; www.sophist.de.

design Editor: Martin Fowler

■

T h o u g h t Wo r k s

■

[email protected]

Yet Another Optimization Article Martin Fowler

T

his is a troubling column to write. I hadn’t planned to write on optimization, because what I have to say has already been said numerous times. Yet, when I give talks, I find there’s still a surprising number of people who don’t know, or at least don’t follow, the advice I’m about to give. So, here goes. (Many of you have probably seen this advice before—my thought to you is to ponder why I need to say this again.) First, performance matters. Although relying on Moore’s law to get us out of slow programs has its merits, I find it increasingly annoying when I get a new version of a program and must upgrade my hardware for it to work acceptably. The question is, “How do we achieve a fast program?” For many programmers, performance is something you pay continuous attention to as you program. Every time you write a fragment of code, you consider the performance implications and code the program to maximize performance. This is an obvious technique—pity it works so badly. Performance is not something you can work on in this way. It involves specific discipline. Some performance work comes from architectural decisions, some from a more tactical optimization activity. But what both share is the fact that it is difficult to make decisions about performance from just looking at the design. Rather, you have to actually run the code and measure performance. 20

IEEE SOFTWARE

May/June 2002

Steps for optimization Optimizing an existing program follows a specific set of steps. First, you need a profiler—a program that can analyze how much time your program spends in its various parts. Software performance has an 80/20 rule: 80 percent of the program’s time is spent on about 20 percent of the code. Trying to optimize performance in the 80 percent of code is futile, so the first order of business is to find that 20 percent of code. Trying to deduce where the program will spend its time is also futile. I know plenty of experienced programmers who always get this wrong. You have to use a profiler. To give the profiler something to chew on, perform some kind of automated run that reasonably simulates the program under its usual conditions. An automated test suite is a good starting point, but make sure you simulate the actual conditions. My colleague Dave Rice has a rule: “Never optimize a multiuser system with single-user tests.” Experience has taught us that a multiuser database system has very different bottlenecks than a single user system—often focused around transaction interactions. The wrong set of tests can easily lead you to the wrong 20 percent of code. Once you’ve found your bottlenecks, you have two choices: speed up the slow things or do the slow things less often. In either case, you must change the software. This is where having a well-designed piece of software really helps. It’s much easier to optimize cohesive, loosely coupled modules. Breaking down a system into many 0740-7459/02/$17.00 © 2002 IEEE

DESIGN

small pieces lets you narrow down the bottlenecks. Having a good automated test suite makes it easier to spot bugs that might slip in during optimization. It’s worth knowing about various optimization tricks, many of which are particular to specific languages and environments. The most important thing to remember is that the tricks are not guaranteed to work—as the saying goes, “Measure twice, cut once.” Unlike a tailor, however, you measure before and after you’ve applied the optimization. Only then do you know if it’s had any effect. It’s revealing how often an optimization has little—or even a negative—effect. If you make an optimization and don’t measure to confirm the performance increase, all you know for certain is that you’ve made your code harder to read. This double measurement is all the more important these days. With optimizing compilers and smart virtual machines, many of the standard optimizing techniques are not just ineffective but also counterintuitive. Craig Larman really brought this home when he told me about some comments he received after a talk at JavaOne about optimization in Java. One builder of an optimizing virtual machine said, in effect, “The comments about thread pools were good, but you shouldn’t use object pools because they will slow down our VM.” Then another VM builder said, “The comments about object pools were good, but you shouldn’t use thread pools because they slow down our VM.” Not only does this reinforce the need to measure with every optimization change, it also suggests that you should log every change made for optimization (a comment tag in the source code is a good option) and retest your optimizations after upgrading your compiler or VM. The optimization you did six months ago could be your bottleneck today. All of this reinforces the key rule that first you need to make you program clear, well factored, and nicely modular. Only when you’ve done that should you optimize.

Some exceptions Although most performance issues are best dealt with in these kinds of optimizations, at times other forms of thinking are important—for example, during early architectural stages, when you’re making decisions that will be costly to reverse later. Again, the only way to really understand these performance issues is to use measurements. In this case, you build exploratory prototypes to perform crude simulations of the environments with which you’re going to work and to get a sense of the relative speeds. It’s tricky, of course, to get a good idea of what the actual environment might be, but then it’s likely that everything you’re working with will be upgraded before you go live anyway. Experiments are still much better than wild guesses. There are also some cases where there are broad rules about slow things. An example I always come across is the slowness of remote procedure calls. Because remote calls are orders of magnitude slower than inprocess calls, it’s important to minimize them, which greatly affects overall design. However, this doesn’t trump measuring. I once came across a situation where people optimize remote methods only to find their bottlenecks were elsewhere. However, minimizing remote calls has proven to be a solid rule of thumb.

If you make an optimization and don’t measure to confirm the performance increase, all you know for certain is that you’ve made your code harder to read.

Some have taken this further, coming up with performance models that you can use to assess different architectural designs. Although this can be handy, it’s difficult to take it too far. It all depends on how good the performance model is, and people usually cannot predict the key factors, even at a broad level. Again, the only real arbiter is measurement. In the end, however, performance is not an absolute. Getting a program to run faster costs money, and it’s a business decision whether to invest in a quicker program. One value of an explicit optimization phase is that the cost of getting fast performance is more explicit, so businesses can trade it against time to market or more features. Much of the reason why I curse at slow software is because it makes good business sense for the builder.

A

s I’ve said, much of this advice— in particular, the advice to write a good clean program first and optimize it later—is well worn. (For a longer description of this approach, read Chapters 28 and 29 of Steve McConnell’s Code Complete, Microsoft Press, 1993.) Good quotes about the perils of premature optimization have been around for over 20 years. The pity is that some people still object when I call the same query method twice in a routine. For everyone who finds nothing new in this column, there exists another challenge—how to make it so there is no need to rewrite it in 10 years.

Martin Fowler is the chief scientist for ThoughtWorks, an Internet systems delivery and consulting company. Contact him at [email protected]. May/June 2002

IEEE SOFTWARE

21

software construction Editors: Andy Hunt and Dave Thomas ■ The Pragmatic Programmers a n d y @ p r a g m a t i c p r o g r a m m e r. c o m ■ d a v e @ p r a g m a t i c p r o g r a m m e r. c o m

Mock Objects Dave Thomas and Andy Hunt

Yet sit and see; Minding true things by what their mockeries be. —Shakespeare, Henry V

O

22

IEEE SOFTWARE

ne thing that makes unit-testing code so hard is the way the real world keeps intruding. If all we had to do was code up tests for methods that sort arrays or generate Fibonacci series, life would be easy. But in the real world we have to test code that uses databases, communications devices, user interfaces, and external applications. We might

drop off, and we all know where that leads. Fortunately there’s a testing pattern that can help. Using mock objects, you can test code in splendid isolation, simulating all those messy real-world things that would otherwise make automated testing impossible. And, as with many other testing practices, the discipline of using mock objects can improve your code’s structure.

have to interface to devices that aren’t yet available or simulate network errors that are impossible to generate locally. This all conspires to stop our unit tests from being neat, self-contained (and orthogonal) chunks of code. Instead, if we’re not careful, we find ourselves writing tests that end up initializing nearly every system component just to give the tests enough context to run. Not only is this time consuming, it also introduces a ridiculous amount of coupling into the testing process: someone changes an interface or a database table, and suddenly the setup code for your poor little unit test dies mysteriously. Even the best-intentioned developers become discouraged after this happens a few times. Eventually, testing starts to

An example: Testing a servlet Servlets are chunks of code that a Web server manages. Requests to certain URLs are forwarded to a servlet container (or manager) such as Jakarta Tomcat (http://jakarta. apache.org/tomcat), which in turn invokes the servlet code. The servlet then builds a response that it sends back to the requesting browser. From the end user’s perspective, it’s just like accessing any other page. Figure 1 shows part of the source of a trivial servlet that converts temperatures from Fahrenheit to Celsius. Let’s quickly step through its operation. When the servlet container receives the request, it automatically invokes the servlet method doGet(), passing in two parameters, a request and a response. (These are important for our testing later). The request parameter contains information about the request; the servlet’s job is to fill in the response. The servlet’s body gets the contents of the field “Fahrenheit” from the request, converts it to Celsius, and writes the results back to the user. The writing is done via a PrintWriter object, which a factory method in the response object provides. If an error occurs converting the number (perhaps the user typed “boo!”

May/June 2002

0740-7459/02/$17.00 © 2002 IEEE

SOFTWARE CONSTRUCTION

into the form’s temperature field), we catch the exception and report the error in the response. Having written this code (or before writing it, for those in the Extreme Programming tribe), we’ll want a unit test to verify it. This is where things start looking difficult. This snippet of code runs in a fairly complex environment (a Web server and a servlet container), and it requires a user sitting at a browser to interact with it. This is hardly the basis of a good automated unit test. But let’s look at our servlet code again. Its interface is pretty simple: as we mentioned before, it receives two parameters, a request and a response. The request object must be able to provide a reasonable string when its getParameter() method is called, and the response object must support setContentType() and getWriter(). It’s starting to look as if we might be able to write some stubs: objects that pretend to be real request and response objects but that contain just enough logic to let us run our code. In principle this is easy: both HttpServletRequest and HttpServletResponse are interfaces, so all we have to do is whip up a couple of classes that implement the interfaces and we’re set. Unfortunately, when we look at the interface, we discover that we’ll need to implement dozens of methods just to get the thing to compile. Fortunately, other folks have already done the work for us. Mock objects Tim Mackinnon, Steve Freeman, and Philip Craig introduced the concept of mock objects in their paper “Endo-Testing: Unit Testing with Mock Objects” (www.cs.ualberta.ca/ ~hoover/cmput401/XP-Notes/xp-conf/ Papers/4_4_MacKinnon.pdf), which they presented at XP2000. Their idea is a natural extension of the ad hoc stubbing that testers have been doing all along. The difference is that they describe a framework to make writing mock objects and incorporating them into unit testing easier. Their paper lists seven good reasons to use a mock object (paraphrased slightly here):

1 public void doGet(HttpServletRequest req, 2 HttpServletResponse res) 3 throws ServletException, IOException 4 { 5 String str_f = req.getParameter(“Fahrenheit”); 6 7 res.setContentType(“text/html”); 8 PrintWriter out = res.getWriter(); 9 10 try { 11 int temp_f = Integer.parseInt(str_f); 12 double temp_c = (temp_f – 32) * 5.0 / 9.0; 13 out.println(“Fahrenheit: “ + temp_f + “, Celsius: “ + temp_c); 14 } 15 catch (NumberFormatException e) { 16 out.println(“Invalid temperature: “ + str_f); 17 } 18 } Figure 1. A trivial servlet that converts temperatures from Fahrenheit to Celsius. ■ ■ ■

■ ■ ■

■

The real object has nondeterministic behavior. The real object is difficult to set up. The real object has behavior that is hard to trigger (for example, a network error). The real object is slow. The real object has (or is) a user interface. The test needs to ask the real object about how it was used (for example, a test might need to check to see that a callback function was actually called). The real object does not yet exist.

Mackinnon, Freeman, and Craig also developed the code for a mock object framework for Java programmers (available at www.mockobjects.com). Let’s use that code to test our servlet. The good news is that in addition to the underlying framework code, the mockobjects package comes with a number of mocked-up application-level objects. You’ll find mock output objects (OutputStream, PrintStream, and PrintWriter), objects that mock the java.sql library, and classes for faking out a servlet environment. In particular, the package provides mockedup versions of HttpServletRequest and HttpServletResponse, which by an incredible coincidence are the types of the parameters of the method we want to test.

We can use mock objects in two distinct ways. First, we can use them to set up an environment in which our test code runs: we can initialize values in the objects that the method under test uses. Figure 2 shows a typical set of tests using the JUnit testing framework, which is available at www.junit. org. We use a MockHttpServletRequest object to set up the context in which to run the test. On line six of the code, we set the parameter “Fahrenheit” to the value “boo!” in the request object. This is equivalent to the user entering “boo!” in the corresponding form field; our mock object eliminates the need for human input when the test runs. Mock objects can also verify that actions were taken. On line seven of Figure 2, we tell the response object that we expect the method under test to set the response’s content type to text/html. Then, on lines 9 and 22, after the method under test has run, we tell the response object to verify that this happened. Here, the mock object eliminates the need for a human to check the result visually. This example shows a pretty trivial verification: in reality, mock objects can verify that fairly complex sequences of actions have been performed. Mock objects can also record the data that was given to them. In our case, the response object receives the May/June 2002

IEEE SOFTWARE

23

SOFTWARE CONSTRUCTION

1 public void test_bad_parameter() throws Exception { 2 TemperatureServlet s = new TemperatureServlet(); 3 MockHttpServletRequest request = new MockHttpServletRequest(); 4 MockHttpServletResponse response = new MockHttpServletResponse(); 5 6 request.setupAddParameter(“Fahrenheit”, “boo!”); 7 response.setExpectedContentType(“text/html”); 8 s.doGet(request, response); 9 response.verify(); 10 assertEquals(“Invalid temperature: boo!\r\n”, 11 response.getOutputStreamContents()); 12 } 13 14 public void test_boil() throws Exception { 15 TemperatureServlet s = new TemperatureServlet(); 16 MockHttpServletRequest request = new MockHttpServletRequest(); 17 MockHttpServletResponse response = new MockHttpServletResponse(); 18 19 request.setupAddParameter(“Fahrenheit”, “212”); 20 response.setExpectedContentType(“text/html”); 21 s.doGet(request, response); 22 response.verify(); 23 assertEquals(“Fahrenheit: 212, Celsius: 100.0\r\n”, 24 response.getOutputStreamContents()); 25 } Figure 2. Mock objects in action—a typical set of tests using the JUnit testing framework.

text that our servlet wants to display on the browser. We can query this value (lines 10 and 23) to check that we’re returning the text we were expecting.

Mock objects It’s likely that we’ve all been using mock objects for years without knowing it. However, it’s also likely that we’ve used them only on an ad

Practical Experience with Mock Objects Nat Pryce My experience is that using mock objects and a test-first methodology forces you to think about design differently from traditional object-oriented design techniques. First, you end up with many small, decoupled classes that are used through composition, rather than inheritance. These classes implement interfaces more than they inherit state and behavior. Second, you think about object interfaces in terms of the services that an object both provides and requires from its environment, making an object’s requirements explicit. This is different from traditional OO design methods that concentrate only on an object’s provided services and try to hide an object’s requirements through encapsulation of private-member variables. Third, you end up thinking more explicitly in terms of interobject protocols. Those protocols are often defined using interface definitions and tested using mock objects before a concrete implementation is produced. Nat Pryce is the technical director at B13media Ltd. Contact him at [email protected]; www.b13media.com.

24

IEEE SOFTWARE

May/June 2002

hoc basis, coding up stubs when we needed them. However, we personally have recently started benefiting from adopting a more systematic approach to creating mock objects. Even things as simple as consistent naming schemes have helped make our tests more readable and the mock objects themselves more portable from project to project. There are several mock object frameworks to choose from. Three for Java are at www.c2.com/cgi/wiki?MockObject, and a fine implementation for Ruby is at www.b13media.com/dev/ruby/ mock.html. If the thought of writing all the mock object classes you might need is intimidating, look at EasyMock (www.easymock.org), a convenient Java API for creating mock objects dynamically. All these implementations are a starting point; you’ll probably need to add new mock object implementations to stub out real objects in your environment. There are also alternatives to mock objects in the servlet environment. In particular, the Jakarta Cactus system (http://jakarta.apache.org/ cactus) is a heavier-weight framework for testing server-side components. Compared to the mock-objects approach, Cactus runs your tests in the actual target environment and tends to produce less fine-grained tests. Depending on your needs, this might or might not be a good thing.

A

funny thing happens when you start using mock objects. As with other low-level testing practices, you might find that your code becomes not only better tested but also better designed and easier to understand (Nat Pryce discusses this in the sidebar). Mock objects won’t solve all your development problems, but they are exceptionally sharp tools to have in your toolbox. Dave Thomas and Andy Hunt are partners in The Pragmatic Programmers, LLC. They feel that software consultants who can’t program shouldn’t be consulting, so they keep current by developing complex software systems for their clients. They also offer training in modern development techniques to programmers and their management. They are coauthors of The Pragmatic Programmer and Programming Ruby, both from Addison-Wesley. Contact them via www.pragmaticprogrammer.com.

focus

guest editors’ introduction

Knowledge Management in Software Engineering Ioana Rus and Mikael Lindvall, Fraunhofer Center for Experimental Software Engineering, Maryland

oftware organizations’ main assets are not plants, buildings, or expensive machines. A software organization’s main asset is its intellectual capital, as it is in sectors such as consulting, law, investment banking, and advertising. The major problem with intellectual capital is that it has legs and walks home every day. At the same rate experience walks out the door, inexperience walks in the door. Whether or not many software organizations admit it, they face the challenge of

S Knowledge management is an expanding and promising discipline that has drawn considerable attention. Recent developments in computer and IT technologies enable sharing documented knowledge independently of time and location. The guest editors provide an overview of these developments to introduce the issue. 26

IEEE SOFTWARE

sustaining the level of competence needed to win contracts and fulfill undertakings. Knowledge management is an emerging discipline that promises to capitalize on organizations’ intellectual capital. The concept of taming knowledge and putting it to work is not new; phrases containing the word knowledge, such as knowledge bases and knowledge engineering, existed before KM became popularized. The artificial intelligence community has long dealt with knowledge representation, storage, and application. Software engineers have engaged in KM-related activities aimed at learning, capturing, and reusing experience, even though they were not using the phrase “knowledge management.” KM is unique because it focuses on the individual as an expert and as the bearer of important knowledge that he or she can systematically share with an organization. KM supports not only the knowhow of a company, but also the know-where,

May/June 2002

know-who, know-what, know-when, and know-why. The KM concept emerged in the mid1980s from the need to derive knowledge from the “deluge of information” and was mainly used as a “business world” term. In the 1990s, many industries adopted the term KM in connection with commercial computer technologies, facilitated by development in areas such as the Internet, group support systems, search engines, portals, data and knowledge warehouses, and the application of statistical analysis and AI techniques. KM implementation and use has rapidly increased since the 1990s: 80 percent of the largest global corporations now have KM projects.1 Over 40 percent of Fortune 1000 companies now have a chief knowledge officer, a senior-level executive who creates an infrastructure and cultural environment for knowledge sharing.2 0740-7459/02/$17.00 © 2002 IEEE

But will KM really help organizations address the problems they face while trying to achieve their business objectives? Or is it just one of those hyped concepts that rise quickly, ambitiously claim to cure organizational headaches, and then fail and fall quietly? KM has drawn widespread attention and has promising features, but it is too soon to say whether or not it will stay. However, we can address other questions: What kind of problems can KM help solve for software organizations? What are its challenges? What are the success factors? How can KM leverage all the knowledge that exists in software organizations? This special issue of IEEE Software addresses these types of questions. Motivation for KM in software engineering Software development is a quickly changing, knowledge-intensive business involving many people working in different phases and activities. The available resources are not increasing along with the increasing needs; therefore, software organizations expect a rise in productivity. Knowledge in software engineering is diverse and its proportions immense and steadily growing. Organizations have problems identifying the content, location, and use of the knowledge. An improved use of this knowledge is the basic motivation and driver for KM in software engineering and deserves deeper analysis. Business needs Organizations can apply KM to provide solutions to pressing business issues. Decreasing time and cost and increasing quality. Organizations constantly need to

decrease software projects’ development time and costs. Avoiding mistakes reduces rework; repeating successful processes increases productivity and the likelihood of further success. So, organizations need to apply process knowledge gained in previous projects to future projects. Unfortunately, the reality is that development teams do not benefit from existing experience and they repeat mistakes even though some individuals in the organization know how to avoid them. Project team members acquire valuable individual experience with each

project—the organization and individuals could gain much more if they could share this knowledge. Making better decisions. In software development, every person involved constantly makes technical or managerial decisions. Most of the time, team members make decisions based on personal knowledge and experience or knowledge gained using informal contacts. This is feasible in small organizations, but as organizations grow and handle a larger volume of information, this process becomes inefficient. Large organizations cannot rely on informal sharing of employees’ personal knowledge. Individual knowledge must be shared and leveraged at project and organization levels. Organizations need to define processes for sharing knowledge so that employees throughout the organization can make correct decisions.

Is KM just one of those hyped concepts that rise quickly, ambitiously claim to cure organizational headaches, and then fail and fall quietly?

Knowledge needs Software organizations have vast amounts of knowledge in various areas, such as knowledge that is critical to achieve business goals. We now look at some of these knowledge areas and organizations’ related needs. Acquiring knowledge about new technologies. The emergence of new technologies

makes software development more efficient, but at the same time, they can be a project manager’s worst nightmare. It is difficult for developers to become proficient with a new technology and managers to understand its impact and estimate a project’s cost when using it. When developers or project managers use a technology that a project’s team members are unfamiliar with, engineers often resort to the “learning by doing” approach, which can result in serious delays. So, organizations must quickly acquire knowledge about new technologies and master them. Accessing domain knowledge. Software development requires access to knowledge not only about its domain and new technologies but also about the domain for which software is being developed. An organization must acquire new domain knowledge either by training or by hiring knowledgeable employees and spreading it throughout the team. May/June 2002

IEEE SOFTWARE

27

Passing knowledge informally is an important aspect of a knowledgesharing culture that should be encouraged.

Sharing knowledge about local policies and practices. Every organization has its own poli-

cies, practices, and culture, which are not only technical but also managerial and administrative. New developers in an organization need knowledge about the existing software base and local programming conventions. Unfortunately, such knowledge typically exists as organizational folklore. Experienced developers often disseminate it to inexperienced developers through ad hoc informal meetings; consequently, not everyone has access to the knowledge they need. Passing knowledge informally is an important aspect of a knowledge-sharing culture that should be encouraged. Nonetheless, formal knowledge capturing and sharing ensures that all employees can access it. So, organizations must formalize knowledge sharing while continuing informal knowledge sharing. Capturing knowledge and knowing who knows what. Software organizations depend

heavily on knowledgeable employees because they are key to the project’s success. However, accessing these people can be difficult. Software developers apply just as much effort and attention determining whom to contact in an organization as they do getting the job done.3 These knowledgeable people are also very mobile. When a person with critical knowledge suddenly leaves an organization, it creates severe knowledge gaps—but probably no one in the organization is even aware of what knowledge they lost. Knowing what employees know is necessary for organizations to create a strategy for preventing valuable knowledge from disappearing. Knowing who has what knowledge is also a requirement for efficiently staffing projects, identifying training needs, and matching employees with training offers. Collaborating and sharing knowledge. Software development is a group activity. Group members are often geographically scattered and work in different time zones. Nonetheless, they must communicate, collaborate, and coordinate. Communication in software engineering is often related to knowledge transfer. Collaboration is related to mutual sharing of knowledge. Group members can coordinate independently of time and space if they can easily access their work artifacts. So, group 28

IEEE SOFTWARE

May/June 2002

members need a way to collaborate and share knowledge independently of time and space. KM background Before we go further into how KM can address software engineering needs, let us introduce some key KM concepts. Knowledge levels The three levels of refinement to knowledge items are data, information, and knowledge. Data consists of discrete, objective facts about events but nothing about its own importance or relevance; it is raw material for creating information. Information is data that is organized to make it useful for end users who perform tasks and make decisions. Knowledge is broader than data and information and requires understanding of information. It is not only contained in information, but also in the relationships among information items, their classification, and metadata (information about information, such as who has created the information). Experience is applied knowledge. Knowledge characteristics KM deals not only with explicit knowledge, which is generally easier to handle, but also tacit knowledge. Explicit knowledge, also known as codified knowledge, is expressed knowledge. It corresponds to the information and skills that employees can easily communicate and document, such as processes, templates, and data. Tacit knowledge is personal knowledge that employees gain through experience; this can be hard to express and is largely influenced by their beliefs, perspectives, and values. The knowledge scope refers to where the knowledge is applicable, to whom it is accessible, and whose activity it supports. Along this dimension, knowledge can be at an individual, group, organization, multiple organization, or industry-wide level. The knowledge evolution cycle defines the phases of organizational knowledge.4 Organizations should have a knowledge management strategy in place for implementing these phases systematically. The phases are 1. Originate/create knowledge. Members of an organization develop knowledge through learning, problem solving, innovation, creativity, and importation from outside sources.

2. Capture/acquire knowledge. Members acquire and capture information about knowledge in explicit forms. 3. Transform/organize knowledge. Organizations organize, transform, or include knowledge in written material and knowledge bases. 4. Deploy/access knowledge. Organizations distribute knowledge through education, training programs, automated knowledge-based systems, or expert networks. 5. Apply knowledge. The organization’s ultimate goal is applying the knowledge—this is the most important part of the life cycle. KM aims to make knowledge available whenever it is needed. Knowledge management An organization’s intellectual capital consists of tangible and intangible assets. Tangible assets, which correspond to documented, explicit knowledge, can vary for different industries and applications but usually include manuals; directories; correspondence with (and information about) clients, vendors, and subcontractors; competitor intelligence; patents; licenses; and knowledge derived from work processes (such as proposals and project artifacts). Intangible assets, which correspond to tacit and undocumented explicit knowledge, consist of skills, experience, and knowledge of an organization’s people. Learning and KM Learning is a fundamental part of KM because employees must internalize (learn) shared knowledge before they can use it to perform specific tasks. Individuals primarily learn from each other, by doing, and through self study. Knowledge also spreads from individuals to groups, and throughout organizations and industries. KM aims to elevate individual knowledge to the organizational level by capturing and sharing individual knowledge and turning it into knowledge the organization can access. Individuals eventually perform tasks to achieve organizational-level goals. Therefore, the iterative knowledge processing and learning activities at the individual level are of utmost importance. As Peter M. Senge says, “Organizations learn only through individuals who learn. Individual learning does not guarantee organizational learning. But without it no organizational learning occurs.”5

KM’s role in software engineering In software development, different approaches have been proposed to reduce project costs, shorten schedules, and increase quality. These approaches address factors such as process improvement, introducing new technologies, and improving people’s performance (“peopleware”). KM addresses mainly peopleware. Because software development is a human and knowledge-intensive creative activity, KM acknowledges the importance of individuals having access to the correct information and knowledge when they need to complete a task or make a decision. KM does not ignore the value or need to address other software development aspects, such as process and technology, nor does it seek to replace them. Instead, it works toward software process improvement by explicitly and systematically addressing the management of organizational knowledge, such as its acquisition, storage, organization, evolution, and effective access. Other software process improvement approaches, such as the Capability Maturity Model, might suggest that knowledge should be managed but do not explicitly state what knowledge needs to be managed and how, when, where, or by and for whom. KM ties together daily production activities, improvement initiatives, and business goals, thereby supporting the establishment of a learning organization. Organizations can view KM as a risk prevention and mitigation strategy, because it explicitly addresses risks that are too often ignored, such as ■ ■

■

■

Organizations can view KM as a risk prevention and mitigation strategy, because it explicitly addresses risks that are too often ignored.

Loss of knowledge due to attrition Lack of knowledge and an overly long time to acquire it due to steep learning curves People repeating mistakes and performing rework because they forgot what they learned from previous projects Individuals who own key knowledge becoming unavailable

Software engineering involves several knowledge types—technical, managerial, domain, corporate, product, and project knowledge. Knowledge can be transferred through formal training or through learning by doing. Formal training is often time conMay/June 2002

IEEE SOFTWARE

29

suming and expensive, and if done externally does not cover local knowledge. Learning by doing can be risky because people continue to make mistakes until they get it right. KM does not replace organized training, but supports it. Documented knowledge can provide the basis for internal training courses based on knowledge packaged as training material. However, KM mainly supports learning by doing. It provides knowledge or pointers to people who have the knowledge, when and where it is needed. KM does not come for free; it requires effort and resources. In KM systems that organizations have implemented so far (see the Experience Factory sidebar and this issue’s feature articles), people other than developers often perform KM activities (such as a chief knowledge officer and his staff, an Experience Factory group, or a software

process improvement group). This supports developers in their daily work instead of loading them with extra effort. Software engineering’s core task is developing software. Documents (such as contracts, project plans, and requirements and design specifications) are also produced during software development. These documents capture knowledge that emerged from solving the project’s problems. Team members can then reuse this knowledge for subsequent projects, for example, by analyzing accepted solutions to different problems. If individuals own knowledge that is not explicitly captured, the organization can leverage that knowledge only if it can identify and access these individuals. Organizations wishing to improve a team’s software engineering capabilities can conduct the task of ensuring that knowledge gained during the project is not lost. They

The Experience Factory Organization Victor R. Basili Carolyn Seaman

synthesizes all experience types, including lessons learned, project data, and technology reports, and provides repository services for this experience. The ExThe basis for the Experience Factory perience Factory employs several methOrganization1 concept is that software ods to package the experience, including development projects can improve their designing measures of various software performance (in terms of cost, quality, process and product characteristics and and schedule) by leveraging experience then building models of these from previous projects. The characteristics that describe concept also takes into account their behavior in different conthe reality that managing this texts. These models’ data come experience is not trivial and Experience Factory Organization from development projects via cannot be left to individual Project Experience factory people, documents, and autoprojects. With deadlines, high organization mated support. expectations for quality and Needs Project When using EFO, not only productivity, and challenging Plan support must the organization add antechnical issues, most developPackaged other suborganization for ment projects cannot devote experience, consulting learning, packaging, and storthe necessary resources to Project Package plan ing experience, but it also must making their experience availExperience (synthesize) Packaged base change the way it does its able for reuse. experience work. An organization adoptThe EFO solves this probRaw experience ing the EFO approach must lem by separating these reDo Analyze Feedback believe that exploiting prior sponsibilities into two distinct experience is the best way to organizations. We call these solve problems and ensure that organizations the Project Orthe development process incorganization, which uses packporates seeking and using this aged experience to deliver Figure A. Flow of information through the experience. The EFO also assoftware products, and the Experience Factory Organization.

30

IEEE SOFTWARE

May/June 2002

Experience Factory, which supports software development by providing tailored experience. Figure A depicts the EFO, which assumes separate logical or physical organizations with different priorities, work processes, and expertise requirements. The Experience Factory analyzes and

can conduct this task during the project and shortly after they complete it. It addresses both acquiring knowledge that was not documented as part of the core activities and analyzing documents to create new knowledge. Included in this task are all forms of lessons learned and postmortem analyses that identify what went right or wrong regarding both the software product and process. These activities also include project data analyses, such as comparisons of estimated and actual costs and effort, planned and actual calendar time, or analysis of change history to reflect project events. These tasks collect and create knowledge about a particular project; any organization can perform them. Although these activities’ results are useful by themselves, they can also be the basis for further knowledge creation and learning. They can be

sumes that the activities of the Experience Factory and those of the Project Organization are integrated. That is, the activities by which the Experience Factory extracts experience and then provides it to projects are well integrated into the activities by which the Project Organization performs its function. Figure A represents this interaction and exchange of experience. Making experience available and usable is crucial but is not the essence of an EFO. “Experience” in an Experience Factory is not only the raw information reported directly from projects. It also includes the valuable results of the analysis and synthesis of that local experience, such as “new” knowledge generated from experience. But the new knowledge is based on applying previous experience on real projects, not on analysis in a vacuum. Thus, an EFO must ■

Package experience by analyzing, synthesizing, and evaluating raw experience and build models that represent abstractions of that experience

stored in repositories and experience bases. At a higher level, organizations and industries must analyze multiple past projects to improve their software developing abilities. This requires extensive knowledge based on many different software development experiences, as well as insights into analysis and synthesis of new knowledge. Patterns, heuristics, best practices, estimation models, and industry-wide standards and recommendations are examples of outcomes from these knowledge-processing activities. We group KM activities that support software development into three categories: by the purpose of their outputs (supporting core SE activities, project improvement, or organizational improvement), the scope of their inputs (documents or data from one or multiple projects), and the effort level required to process inputs to serve SE needs. We use this classification to describe how both existing and new

Maintain an experience base or repository of data, experience, models, and other forms of knowledge and experience ■ Support projects in identifying and using the appropriate experiences for the situation ■

Victor Basili first presented the EFO concept in a keynote address at COMPSAC in 1989.2 This was before the term “knowledge management” became popular, but the EFO addresses many of the same concerns. This learning-organization concept evolved from our experiences in the NASA Software Engineering Laboratory, a joint effort of the University of Maryland, Computer Sciences Corporation, and the NASA Goddard Space Flight Center. The SEL’s high-level goal was to improve Goddard’s software processes and products. The application of EFO ideas resulted in a continuous improvement in software quality and cost reduction during the SEL’s quarter-century lifespan.3 Measured over three baseline periods in 1987, 1991, and 1995 (each baseline was calculated based on about three years’ worth of

data), demonstrated improvements included decreases in development defect rates of 75 percent between the 1987 and 1991 baselines and 37 percent between the 1991 and 1995 baselines. We also observed reduced development costs between subsequent baselines of 55 percent and 42 percent, respectively.

References 1. V.R. Basili and G. Caldiera, “Improve Software Quality by Reusing Knowledge and Experience,” Sloan Management Rev., vol. 37, no. 1, Fall 1995, pp. 55–64. 2. V.R. Basili, “Software Development: A Paradigm for the Future,” Proc. 13th Int’l Computer Software and Applications Conf. (COMPSAC 89) IEEE CS Press, Los Alamitos, Calif., 1989, pp. 471–485. 3. V.R. Basili et al., “Special Report: SEL’s Software Process-Improvement Program,” IEEE Software, vol. 12, no. 6, Nov./Dec. 1995, pp. 83–87.

Victor R. Basili’s biography appears on page 49.

Carolyn Seaman is an assistant professor of Information Systems at the University of Maryland, Baltimore County and a research scientist at the Fraunhofer Center for Experimental Software Engineering, Maryland. Contact her at cseaman@ umbc.edu.

May/June 2002

IEEE SOFTWARE

31

A common problem that knowledge management addresses is expert identification.

software engineering processes and tools can fit into a KM strategy. In the remainder of this section, we also discuss resources external to the organization that are built around the notion of capturing and sharing knowledge. Supporting core SE activities Document management, competence management, and software reuse are knowledge management activities that support software development. Document management. A software development project involves a variety of document-driven processes and activities. The work frequently focuses on authoring, reviewing, editing, and using these documents, which become the organization’s assets in capturing explicit knowledge. Therefore, document management is a basic activity toward supporting an organization’s implementation of a knowledge management system. DM systems enable employees throughout the organization to share documented knowledge. Many commercial tools support DM, such as Hyperwave, Microsoft Sharepoint, Lotus Domino, and Xerox DocuShare (see “The Experience Factory Organization” sidebar), and include features such as defining process workflows and finding experts. Competence management and expert identification. Far from all of an organization’s tacit

knowledge can be made explicit, and far from all explicit knowledge can be documented. So, an organization must track who knows what to fully utilize undocumented knowledge. An elaborate solution to this problem is competence management, or skills management. As we noted earlier, a common problem that knowledge management addresses is expert identification. Competence management systems, such as SkillScape and SkillView, include tools that let experts generate and edit their own profiles. Other tools, such as KnowledgeMail, automatically generate competence profiles by assuming that peoples’ emails and documents reflect their expertise. These tools analyze email repositories and documents and build keyword-based profiles that characterize each employee. Software organizations can use them to identify experts in various technical areas, such as specific programming languages, database technologies, or operating systems. 32

IEEE SOFTWARE

May/June 2002

Software reuse. There are endless stories about how programmers continuously implement the same solutions in slightly different ways. Software reuse approaches attempt to reduce this rework by establishing a reuse repository. Programmers submit software they believe would be useful to others. The software development process must change so that instead of developing all software from scratch, a programmer first searches the repository for reusable parts. Only if the programmer found nothing useful would he or she write the software from scratch. This same concept can apply to all software engineering artifacts.

Supporting product and project memory Learning from experience requires a product and project memory. The environment in which software engineers conduct their daily work often supports creating such a memory. Version control, change management, documenting design decisions, and requirements traceability are software engineering practices that help build such memories as a direct or side effect of software development. Version control systems, such as the Source Code Control System, represent a class of tools that indirectly create a project memory. Each version of a document has a record attached with information about who made the change, as well as when and why they made it. This “memory” indicates the software’s evolution. Software engineers use this information for advanced analysis of software products and processes.6 They can also use the who-changed-it information to identify experts. Design Rationale is an example of an approach that explicitly captures software design decisions to create a product memory and help avoid repeating mistakes.7 This is important because during design, engineers test different technical solutions and make decisions on the basis of these test results. Unfortunately, these decisions are rarely captured, making it difficult for anyone else to understand the reasons behind the solutions. Software requirements drive the development of software systems, but the connection between the final system and its requirements is fuzzy. Traceability is an approach that explicitly connects the requirements and the final software system. It indirectly contributes to the product memory and helps answer

Useful URLs ACM Special Interest Groups: www.acm.org/sigs/guide98.html CeBASE: www.cebase.org Carnegie Mellon’s Software Process Improvement Network: www.sei.cmu.edu/collaborating/spins Expert Exchange: www.experts-exchange.com Fraunhofer Center for Experimental Software Engineering, Maryland: http://fc-md.umd.edu Hyperwave: www.hyperwave.com IBM’s Lotus: www.lotus.com/home.nsf/welcome/km KnowledegeMail: www.knowledgemail.com Microsoft Product Support Services: http://search.support.microsoft.com/kb Microsoft Sharepoint Technologies: www.microsoft.com/sharepoint Oracle Support Services: www.oracle.com/support/index. html?content.html Perl’s FAQ: www.perl.com/pub/q/faqs Skillscape: www.skillscape.com Skillview: www.skillview.com Software Engineering Body of Knowledge: www.swebok.org Software Program Managers Network: www.spmn.com Sun’s Java community: http://developer.java.sun.com/ developer/community ViSEK at Fraunhofer Center for Experimental Software Engineering, Germany: www.iese.fhg.de/Projects/ViSEK Xerox Docushare: http://docushare.xerox.com

questions such as “What requirements led to a particular piece of source code?” and “What code did the engineers develop to satisfy this particular requirement?” Supporting learning and improvement Project managers need to make a series of decisions at the beginning of and during projects. Typically, their personal experience and “gut feeling” guide their decisions. But because software development is such a complex and diverse process, gut feelings might not be sufficient, and not all managers have extensive experience. For these reasons, predictive models can guide decision making for future projects based on past projects. This requires having a metrics program in place, collecting project data with a well-defined goal in a metrics repository, and then analyzing and processing the data to generate predictive models. Managers or process improvement personnel analyze, synthesize, and process input data using different methods, depending on the model’s purpose and the input and output type. Analytical models have as input data from a large number of projects (either numerical data or qualitative data converted into quantitative levels). Using these formulas for the data that characterize a new project, project managers can estimate cost, effort, defects, reliability, and other product and project parameters. Building, using, and improving these models become a natural part of the KM strategy. Another class of predictive models that captures the development process structure and the internal-process variable relationships consists of executable behavioral process models and simulators based on system dynamics8 and discrete event modeling. Engineers and managers can execute these models, simulate what-if scenarios, and analyze possible outcomes for multiple decisions. These models capture knowledge that addresses process structure and behavior. The prediction quality these models offer depends on the collected data quality. Project information can also be in a qualitative form (such as cases and lessons learned, success and failure stories, and problems and corresponding solutions) represented in formats such as rules, indexed cases, or semantic networks. Applying induction, generalization, and abstraction on

this knowledge can generate new knowledge (manually, or automatically by applying AI techniques) applicable to similar future problems in similar contexts. This is how patterns, best-practice guidelines, handbooks, and standards are derived. The models transform raw point data into explicit and more applicable knowledge. For example, case based-systems capture project experiences to accommodate software development process diversity while retaining a level of discipline and standards. These experiences provide developers with knowledge of an organization’s previous development issues. Deviations from the standard process are opportunities to improve the process itself. Deviations also work as cases in the experience base. As organizations acquire more experience in the form of cases, the development process is iterated and becomes even more refined. Implementing KM Implementing KM involves many challenges and obstacles. Three issues are particularly important:1 ■

Technology issues. Software technology supports KM, but it is not always possiMay/June 2002

IEEE SOFTWARE

33

An analysis of KM failures reveals that many organizations who failed did not determine their goals and strategy before implementing KM systems.

■

■

ble to integrate all the different subsystems and tools to achieve the planned level of sharing. Security is a requirement that the available technology does not often provide satisfactorily. Organizational issues. It is a mistake for organizations to focus only on technology and not on methodology. It is easy to fall into the technology trap and devote all resources to technology development, without planning for KM implementation. Individual issues. Employees often do not have time to input or search for knowledge, do not want to give away their knowledge, and do not want to reuse someone else’s knowledge.

An analysis of KM failures reveals that many organizations who failed did not determine their goals and strategy before implementing KM systems. In fact, 50 to 60 percent of KM deployments failed because organizations did not have a good KM deployment methodology or process, if any.1 Some organizations ended up managing documents instead of meaningful knowledge. This is an easy mistake to make, because many tools advertised as KM tools address document management rather than knowledge management. Lightweight KM approaches A problem with KM is that it might take time before measurable benefits appear. It generally takes a long time before knowledge bases contain a critical mass of knowledge. However, organizations can quickly and easily implement lightweight KM approaches— this can pay off quickly while laying the foundation for long-term goals. One such tool is the Knowledge Dust Collector,9 which supports peer-to-peer knowledge sharing by capturing knowledge that employees exchange and use every day. The knowledge “dust” evolves over time into knowledge “pearls,” a refined knowledge form. The Knowledge Dust Collector captures dialogs regarding technical problems (knowledge dust), analyzes them, and turned them into frequently asked questions (knowledge pearls). These FAQs are then analyzed and turned into best practices (extended knowledge pearls). Organizational culture Although new technology makes sharing

34

IEEE SOFTWARE

May/June 2002

knowledge easier than ever, organizational cultures might not promote it. Some cultures even overly encourage individualism and ban cooperative work. Lack of a “knowledge culture” has been cited as the number one obstacle to successful KM.4 If organizations don’t foster a sharing culture, employees might feel possessive about their knowledge and won’t be forthcoming in sharing it. Employees know that the organization values them because of their knowledge; they might fear that they will be considered redundant and disposable as soon as the employer has captured their knowledge. Employees might not be willing to share negative experiences and lessons learned based on failures because of their negative connotation. So although KM’s purpose is to avoid similar mistakes, employees might fear that such information could be used against them. Another hurdle is the “not invented here” syndrome—some believe that software engineers are reluctant to reuse other people’s solutions. Although change is hard, such beliefs must be revisited and replaced by a positive attitude that engenders and rewards sharing. KM champions Hewlett-Packard advocates using an evangelist or champion for any KM initiative—someone who encourages employees to contribute and use the system and who is always its proponent.10 Reward systems Organizations must not only encourage but also reward employees who share their knowledge, search for knowledge, and use others’ knowledge. To encourage sharing and reusing knowledge, Xerox recommends creating a “hall of fame” for those people whose contributions have solved real business problems. Xerox rewards staff that regularly share useful information and identifies them as key contributors to the program. Bruce Karney, evangelist of a Hewlett-Packard KM initiative, gave out free Lotus Notes licenses and free airline miles to prospective users.10 Infosys rewards employee contribution and use of knowledge with “knowledge currency units,” which they can convert into cash. The online expertise provider ExpertExchange rewards experts with points for answering questions and recognizes those with the most points on the front page of their Web site.

Leveraging employees’ expertise Most approaches that support experience reuse and knowledge management assume that all relevant experience can be collected and recorded, but in practice, this does not hold true. A variety of KM solutions address different aspects and tasks of this problem. For example, Ericsson Software Technology implemented a version of the Experience Factory (see the related sidebar) called the Experience Engine.11 Instead of relying on experience stored in experience bases, the Experience Engine relies on tacit knowledge. Ericsson created two roles to make the tacit knowledge accessible to a larger group of employees. The experience communicator is an employee who has in-depth knowledge on one or more topics. The experience broker connects the experience communicator with the employee who owns the problem. The communicator should not solve the problem but instead educate the problem’s owner on how to solve it. The German organization sd&m has successfully implemented a similar approach.12 Relying on tacit rather than explicit knowledge is appealing because it relaxes the requirement to extensively document knowledge. However, although this approach uses the knowledge, it still does not solve the problem of an organization being dependent on its employees and their tacit knowledge. The benefit of explicit knowledge or experience is that it can be stored, organized, and disseminated to a third party without the originator’s involvement. The drawback is that organizations spend considerable effort to produce explicit knowledge. Some development practices, such as pair programming, facilitate knowledge sharing between peers, while job rotation helps knowledge spread throughout the project or organization. Software organizations should encourage these habits to create a knowledge-sharing culture. To achieve the maximum benefit from knowledge sharing, organizations should encourage employees to document and store their knowledge in a KM repository whenever they help another employee. In doing so, they ensure that the information they passed on is recorded and will help other employees, because what is a problem for one can also be a problem for many.

The most used form of knowledge sharing probably occurs in communities in which members can come together and share knowledge and experience. Communities are popular because they are relatively easy to form. Software organizations have formed numerous useful communities. Some examples include the Software Program Managers Network; the Software Experience Consortium, for companies seeking to share experience; Sun’s community for Java programmers, the Software Process Improvement Network; and special interest groups of the IEEE and ACM. Leveraging organizational expertise Organizations strive to learn more and leverage their expertise through input that comes from outside the organization.

The benefit of explicit knowledge or experience is that it can be stored, organized, and disseminated to a third party without the originator’s involvement.

Sharing experience with customers. Organizations learn not only from their own experiences but also from external sources, typically technology vendors. Several software vendors provide online knowledge bases, such as Microsoft’s Knowledge Base, Oracle’s Support Center, and Perl’s FAQ. Such knowledge bases are often open to the public and let software engineers search for knowledge. These knowledge bases resulted from capturing vendor representatives’ product knowledge and making it available to the vendors’ customers. Industry-wide knowledge sharing and education. At the software industry level, com-

mittees or groups of experts identify patterns (such as software design patterns) and generate handbooks and standards (such as those from the IEEE and ISO) that are generally applicable to software development, to leverage the experience and knowledge of all software development organizations. This is not something any individual or organization can perform, because it takes much effort and requires a considerable amount of SE knowledge and access to project data. An example of the numerous industrywide knowledge initiatives is the Software Engineering Body of Knowledge (SWEBOK). It defines the knowledge that a practicing software engineer needs to master on a daily basis. Another example is ISO15504, a comprehensive collection of software engineering knowledge that describes processes related to SE. May/June 2002

IEEE SOFTWARE

35

There are good reasons to believe that KM for software engineering can succeed if organizations appropriately focus and implement it.

Projects whose goal is to build knowledge bases include the Center for Empirically Based Software Engineering and ViSEK (Virtual SE Competence Center). They accumulate empirical models to provide validated guidelines for selecting techniques and models, recommend areas for research, and support SE education. KM challenges As we noted earlier, implementing KM is challenging because many resources and much time and effort are required before benefits become visible. Project managers who feel they need to focus on completing their current project on time, and not on helping the next project manager succeed, often consider this a burden. Another obstacle is that most SE knowledge is not explicit. Organizations have little time to make knowledge explicit. Additionally, there are few approaches and tools that turn tacit into explicit knowledge. Technology’s fast pace often discourages software engineers from analyzing the knowledge they gained during the project, believing that sharing the knowledge in the future will not be useful. Opportunities Despite the challenges we presented earlier, there are good reasons to believe that KM for software engineering can succeed if organizations appropriately focus and implement it. A major argument for KM is that software organizations should already have much of the appropriate information technology in place to support KM systems. IT might be intimidating to many people, but not to software engineers. Instead, we expect that engineers will benefit even more from advanced technology. Additionally, all artifacts are already in electronic form and thus can easily be distributed and shared. In fact, software engineers already share knowledge to a large degree in some environments. A good example is Google, whose user community of software engineers share knowledge by answering questions and helping solve problems that other software engineers post, without being compensated. This shows that software engineers are willing to share their knowledge, even outside their company, and that capturing gained knowledge is worth the effort even though technology changes quickly.

36

IEEE SOFTWARE

May/June 2002

State of the practice This special issue of IEEE Software investigates KM’s state of the practice in software engineering. The large number of submissions we received (over 40 papers presenting SE applications of KM) shows the interest for this topic throughout the software community. The selected articles report on the needs, implementations, issues, results, success factors, and lessons learned from a variety of KM applications. These articles cover a diverse ground; they come from the US, Europe (Germany, Norway, and Finland), and Asia (India and Taiwan) and from commercial and government organizations developing either software for diverse domains (satellites, cars, electronics, telecommunications, and the Internet) or integrated circuits. The authors are practitioners, consultants, researchers, and KM officers. The articles address different levels and scopes of KM activities, from project-level knowledge (captured by postmortem analysis or by gathering knowledge on demand), to organization-wide initiatives. They examine various techniques through case studies, from local analysis to traceability, to complex and highly automated knowledge and experience repositories. Jay Liebowitz’s “A Look at NASA Goddard Space Flight Center’s Knowledge Management Initiatives” describes the creation of a NASA knowledge management team, a KM officer position, and a series of KM initiatives at NASA Goddard Space Flight Center. They learned that KM should start small and see what works in a specific environment and that knowledge should be collected during projects, not after their completion. They also learned to capture individual knowledge through interviews before people leave the organization and to embed KM processes in daily activities. Andreas Birk, Torgeir Dingsøyr, and Tor Stålhane’s “Postmortem: Never Leave a Project without It” addresses the problem that the individual knowledge that engineers gain during projects is not reused or shared between teams. Birk and his colleagues’ solution was to use systematic postmortem analysis for capturing and reusing experience and improvement suggestions. Having an open atmosphere for the project team to discuss issues was a key success factor. As a result, teams experienced increased

understanding and experience sharing, identification of potential improvement needs, and increased job satisfaction. Software development and acquisition competencies are a scarce resource in many companies, and DaimlerChrysler is no exception. To cope with this, DaimlerChrysler researchers set up a Software Experience Center project. SEC explicitly reuses experience from previous software projects using a customized Experience Factory approach (see related sidebar). Kurt Schneider, Jan-Peter von Hunnius, and Victor R. Basili’s “Experience in Implementing a Learning Software Organization” presents the most important challenges they faced: learning implies change—and change is not easy. Learning involves risks that must be mitigated, and experience reuse requires packaging, which is not trivial. Success factors included understanding that change takes time and that experience must be packaged and tailored for the organization. Thanks to the SEC project, learning has become part of developers’ daily routine, leading to process improvements. From studying best practices in 30 system development organizations, Balusubramaniam Ramesh observed the need to link knowledge fragments (which were spread across the organization) to facilitate successful knowledge transfer and reuse. In “Process Knowledge Management with Traceability,” he proposes creating, storing, retrieving, transferring, and applying knowledge by using traceability (creating and maintaining relationships between objects and people in software development). This is possible where development of facilities that support KM processes (traceability schemes and tools) exist. This approach’s benefits are enhanced collaboration and communication, new knowledge creation, knowledge retention, and increased knowledge availability and access for software developers. In Shivram Ramasubramanian and Gokulakrishnan Jagadeesan’s article, “Knowledge Management at Infosys,” Infosys, an Indian consultancy and software services company, realized that locating information and knowledge needed for project activities took too long. Senior management therefore launched a company-wide initiative for building and maintaining a KM infrastructure based on central repositories of documents and expertise maps. They introduced an incentive sys-

tem that converted “knowledge units” into cash to encourage use of and contributions to these repositories. The initiative resulted in reduced defects, increased productivity, decreased cost (mainly by reducing mistakes and rework), and better teamwork. The problem of retaining, reusing, and transferring knowledge occurs not only in SE but also in other knowledge-intensive domains. Borrowing solutions and experience from these other domains can benefit SE. Chin-Ping Wei, Paul Jen-Hwa Hu, and Hung-Huang Chen’s “Design and Evaluation of a Knowledge Management System” reports on KM implementation in an integrated-circuit assembly and testing organization. The company implemented a system that supports expert identification and collaboration for when engineers need to solve a specific problem. Once a problem is solved, its history and solution are stored in a codification-based knowledge repository for later reuse. This system leads to ease of locating experts and sharing knowledge, increases productivity, reduces production interruptions caused by lack of knowledge and search for knowledge, and reduces delayed responses to customer inquiries or complaints. Seija Komi-Sirviö, Annukka Mäntyniemi, and Veikko Seppänen in “Toward a Practical Solution for Capturing Knowledge for Software Projects,” emphasize the need for addressing local needs, problems, and specific context for KM initiative implementation. At an organization that develops softwareintensive electronic products, KM activities were not as efficient as they expected. They replaced their current large scale, technology-centered KM approach with a project need-based approach to knowledge collection and delivery. This proved successful, leading to project staff satisfaction.

Recent developments in IT definitely enable sharing documented knowledge independent of time and space.

N

ot all KM initiatives presented here (maybe none of them) address all aspects of KM. This is normal because KM has not matured yet, especially in SE. When an organization implements such a program, it has to start small. It needs to identify which specific problems and priorities it should address and what does or does not work for that organization. The results must show benefits relatively quickly and convince May/June 2002

IEEE SOFTWARE

37

both the employees to use the system and the management to support it (by continuous investment in the necessary resources). It is too early to say whether knowledge management in software engineering will survive, but it is clear that software organizations struggle to retain valuable knowledge and they still depend on knowledge possessed by their employees. From the enthusiastic response to this special issue, reflected both by the large number of high quality articles submitted (many of them presenting implementation of KM in industry) and by the impressive number of volunteering reviewers, we infer that KM is of interest to many software professionals and researchers and is being tested by many companies as a potential solution to these problems. Recent developments in IT definitely enable sharing documented knowledge independent of time and space. We foresee that in the future there will also be support for capturing and disseminating knowledge in various formats, enabling organizations and individuals to share knowledge on a worldwide scale.

About the Authors Ioana Rus is a scientist at Fraunhofer Center for Empirical Software Engineering, Mary-

land. Her research interests include experience and knowledge management, software process modeling and simulation, process improvement, measurement, and empirical studies in software development. She has a PhD in computer science and Engineering and is a member of the IEEE Computer Society and ACM. Contact her at [email protected].

Mikael Lindvall is a scientist at Fraunhofer Center for Experimental Software Engineering, Maryland. He specializes in work on experience and knowledge management in software engineering. He is currently working on ways of building experience bases to attract users to both contribute and use experience bases. He received his PhD in computer science from Linköpings University, Sweden. His PhD work was based on a commercial development project at Ericsson Radio and focused on the evolution of object-oriented systems. Contact him at [email protected].

38

IEEE SOFTWARE

May/June 2002

Acknowledgments We thank Tom McGibbon and David Nicholls from Data Analysis Center for Software, who supported our work when writing two knowledge management state-of-the-art reports on which we based this tutorial.14,15 We thank Seija Komi-Sirvio, Patricia Costa, Scott Henninger, Raimund Feldman, Forrest Shull, and Rose Pajerski for reviewing; Kathleen Dangle and Tricia Larsen for useful discussions on KM and software process improvement; and Jennifer Dix for proofreading. Many thanks go to the IEEE Software editorial board who anticipated the benefit of this special issue, and to the reviewers of the articles who kindly offered their help for improving the quality of the published material.

References 1. G. Lawton, “Knowledge Management: Ready for Prime Time?” Computer, vol. 34, no. 2, Feb. 2001, pp. 12–14. 2. D.E. O’Leary, “Enterprise Knowledge Management,” Computer, vol. 31, no. 3, Mar. 1998, pp. 54–61. 3. D.E. Perry, N. Staudenmayer, and L. Votta, “People, Organizations, and Process Improvement,” IEEE Software, vol. 11, no. 4, July/Aug. 1994, pp. 36–45. 4. W. Agresti, “Knowledge Management,” Advances in Computers, vol. 53, Academic Press, London, 2000, pp. 171–283. 5. P.M. Senge, The Fifth Discipline: The Art and Practice of the Learning Organization, Currency Doubleday, New York, 1990, p. 139. 6. S.G. Eick et al., “Does Code Decay? Assessing the Evidence from Change Management Data,” Trans. Software Eng., vol. 27, no. 1, Jan. 2000, pp. 1–12. 7. C. Potts and G. Bruns, “Recording the Reasons for Design Decisions,” Proc. 10th Int’l Conf. Software Eng. (ICSE 88), IEEE CS Press, Los Alamitos, Calif., 1988, pp. 418–427. 8. T. Abdel-Hamid and S.E. Madnick, Software Project Dynamics: An Integrated Approach, Prentice-Hall, Englewood Cliffs, N.J., 1991. 9. M. Lindvall et al., “An Analysis of Three Experience Bases,” Proc. Workshop on Learning Software Organizations. (LSO 01), Springer-Verlag, Hiedelberg, Germany, pp. 106–119. 10. T. Davenport, Knowledge Management Case Study, Knowledge Management at Hewlett-Packard, 2002, www.bus.utexas.edu/kman/hpcase.htm. 11. C. Johansson and P.C.M. Hall, “Talk to Paula and Peter—They Are Experienced,” Proc. Workshop on Learning Software Organizations (LSO 99), SpringerVerlag, Hiedelberg, Germany, 1999, pp. 171–185. 12. P. Brössler, “Knowledge Management at a Software Engineering Company—An Experience Report,” Proc. Workshop Learning Software Organizations (LSO 99), Springer-Verlag, Hiedelberg, Germany, 1999, pp. 163–170. 13. T. Davenport, Knowledge Management Case Study, Knowledge Management at Microsoft, 2002, www.bus.utexas.edu/kman/microsoft.htm. 14. M. Lindvall, Software Tools for Knowledge Management, tech. report, DoD Data Analysis Center for Software, Rome, N.Y., 2001. 15. I. Rus, M. Lindvall, and S. Sinha, Knowledge Management in Software Engineering, tech. report, DoD Data Analysis Center for Software, Rome, N.Y., 2001.

focus

knowledge management

A Look at NASA Goddard Space Flight Center’s Knowledge Management Initiatives Jay Liebowitz, NASA Goddard Space Flight Center

s we move from the information age to the knowledge era, knowledge is being treated as a key asset. Organizations try to maintain their competitive edge by sharing knowledge internally with their employees and externally with their customers and stakeholders. NASA and others are realizing that to maximize their organizations’ potential, they must fully leverage their knowledge or intellectual capital.1 To accomplish this task, an emerging trend is to develop knowledge

A Knowledge management offers the public sector and nonprofits ways to improve practices and processes, enhance employee capabilities, improve customer service, and decrease costs. NASA Goddard Space Flight Center has several knowledge management initiatives underway. 40

IEEE SOFTWARE

management and knowledge-sharing initiatives within organizations. For example, NASA formed a Knowledge Management Team, comprised of NASA representatives. Government agencies are turning to knowledge management as they face downsizing and the retirement of half their workforce in the next few years.1 The US Navy has spent US$30 billion to transform itself into a knowledge-centric organization.1 The General Services Administration (GSA) sees knowledge management as the foundation for electronic government.1 The Federal Aviation Administration has embraced knowledge management and finds that communities are the primary resource for transfer of tacit knowledge to explicit knowledge.1 The Social Security Administration feels that subject matter experts are beginning to share best practices readily through its SSA intranet, PRIDE (Project Resource guIDE).1

May/June 2002

Many of these organizations are reaping benefits from their efforts, including retaining the expertise of personnel, increasing customer satisfaction, supporting e-government initiatives, and increasing innovation. Knowledge management offers the public sector and nonprofits ways to improve practices and processes, enhance employee capabilities, improve customer service, and decrease costs. Knowledge management initiatives NASA Goddard Space Flight Center (GSFC) has several knowledge management initiatives under way on the expert and knowledge retention side, including ■

An innovative best practices and lessonslearned CD, based on an ongoing satellite mission called MAP, developed in cooperation with NASA’s Academy for 0740-7459/02/$17.00 © 2002 IEEE

■ ■

■

■

■ ■ ■

■

■

■ ■

Program and Project Leadership Narrative case studies on GSFC’s successful and problematic missions Launching a knowledge preservation pilot project to capture stories of GSFC personnel in the program and project management and systems engineering areas via online Web-searchable video Enhancing the NASA Lessons Learned Information System (see http://llis.nasa. gov) including a “push” capability to send appropriate lessons-learned to individuals based on completed user profiles and designing the next-generation LLIS Conducting knowledge sharing forums in the project management, engineering, and scientific areas Exploring learning and knowledge sharing proficiencies Building an expertise locator (myExperts) via the GSFC intranet portal Building codification and Web access to explicit knowledge via a historical database of documents relating to various GSFC programs and projects Spawning online communities at GSFC through the Process-Based Mission Assurance (PBMA) Knowledge Management System (see http://pbma.hq.nasa.gov) Capturing knowledge via exit interview packets and tapping the knowledge of the GSFC Retirees and Alumni Association Establishing “Creative Learning Groups” at GSFC Discussing a Knowledge Recognition Day for GSFC

In addition to these initiatives, and to provide knowledge stewardship roles, GSFC has formed the Knowledge Management Working Group, chaired by the Knowledge Management Officer and comprised of representatives from each directorate, human resources, library, public affairs, and the CIO office. One knowledge management project that has direct implications for GSFC software development activities involves the Knowledge Preservation Pilot Project. This project captures video nuggets of key GSFC experts who “tell” their critical stories and lessonslearned involving the systems engineering area (including software development) and program and project management. These videos are made available over the Web

Selected Bibliography ■ T. Davenport and J. Beck, The Attention Economy, HBS Press, Cam■ ■ ■ ■

■

bridge, Mass., 2001. J. Liebowitz (ed.), The Knowledge Management Handbook, CRC Press, Boca Raton, Fla., 1999. J. Liebowitz and T. Beckman, Knowledge Organizations: What Every Manager Should Know, St. Lucie Press, Boca Raton, Fla., 1998. J. Liebowitz, Building Organizational Intelligence: A Knowledge Management Primer, CRC Press, Boca Raton, Fla., 2000. B. Montano et al., “SMARTVision Methodology: A Methodology for Knowledge Management,” J. Knowledge Management, vol. 5, no. 4, 2001. M. Rumizen, The Complete Idiot’s Guide to Knowledge Management, PWS Publishing, Boston, Mass., 2002.

(currently, these video nuggets are captured in the PBMA Web resource). We are still exploring the best video indexing, ingesting, and search and retrieval software (for example, Virage, Convera, and so on) for the online Web searchable video component. We expect to share valuable lessons-learned to avoid reinventing the wheel and to heighten the potential for future project success. Additionally, we are developing metrics to help determine the value-added benefits of using knowledge management. Measuring the success of knowledge management initiatives is an important part of GSFC’s knowledge management implementation plan. We are considering metrics from the software engineering and intellectual asset management areas, including innovation, productivity, people retention, knowledge retention, and return on vision (“results and mission success” in NASA language) measures. Quantifying intangible assets is difficult, but hopefully the knowledge management and software engineering communities can collaborate and address this critical area. Lessons learned According to Rosina Weber and her colleagues’ study,2 most lessons-learned programs are ineffective. This is partly because of reliance on a “passive” information collection, analysis, and dissemination approach versus an “active” one. The enhanced version of the NASA LLIS that GSFC operates includes a “push” feature based on user profiles that disseminates appropriate lessons May/June 2002

IEEE SOFTWARE

41

About the Author Jay Liebowitz is the knowledge management officer for the NASA Goddard Space Flight Center. His research interests are expert/intelligent systems, knowledge management, and knowledge audits. He holds a Doctor of Science from George Washington University. Contact him at NASA Goddard Space Flight Center, Code 300, Greenbelt, MD 20771; liebowit@ pop300.gsfc.nasa.gov.

management team across the various business units as a steering committee. From a technical viewpoint, we have also learned some valuable lessons, including ■

■

(matching the user’s profile) to individuals who can benefit from these lessons. Artificial intelligence technology, via intelligent agents, could improve the utility of lessonslearned programs, for example, by providing customized search capabilities. Various lessons-learned can be gleaned from knowledge management efforts. Try a lessons-learned pilot before scaling up to an enterprise-wide solution. You should carefully consider people, culture, technology, communities of practice, costs, and perceived power shift issues. Consider having a lessons-learned component before a project is finally finished. Specifically, enter a set of lessons-learned from a given project into your organization’s lessons-learned repository before the project is officially completed. Conduct exit interviews to glean a set of lessons-learned from the individual before he or she leaves the organization. For a lessons-learned program about knowledge management to work well, in general, incentives must be strengthened to reuse knowledge. Tying the contribution and application of knowledge to performance and review criteria (for example, merit pay) might help encourage a knowledge-sharing environment. For substantial projects, document and encode lessons-learned as standard practice. Include the name of the individual associated with lessons and tips to give him or her greater recognition, as done in the Eureka system at Xerox. American Management Systems publishes Best Knews, a newsletter listing the most frequently used nuggets of knowledge from their lessons-learned repository. Systematically rotating jobs provides vocational stimulation and awareness of the entire system process, helping in knowledge reuse and understanding. Lastly, having a senior-level management official spearhead these initiatives is useful for developing a knowledge 42

IEEE SOFTWARE

May/June 2002

■

You need an integrated knowledge management architecture to cut across the functional silos. Knowledge management processes must be embedded within the daily activities of the employee for knowledge management to be successful. Continue research to advance the state of the art in knowledge management.

Some organizations still might not be ready for knowledge management because of budgetary constraints, cultural misalignments, short-term interests versus longevity goals, and history (being burned before on management fads or the hot management concept of the day). Knowledge management might not be for everyone, but at least awareness can be created. Hopefully, if these lessons are applied, an organization will have a greater likelihood to succeed in knowledge management, resulting in improved organizational morale, increased innovation, and better customer and stakeholder relations.

T

he years ahead look promising for organizations that make learning a priority. Knowledge management is putting us on the road to getting there, but it’s only a start. Hopefully, senior management throughout industry and government will embrace these concepts as the intangibles become tangible.

References 1. M. Eisenhart, “Knowledge Management in the Public Sector, Washington’s Need to Know,” Knowledge Management Magazine, vol. 4, no. 1, Jan. 2001. 2. R. Weber et al., “Intelligent Lessons Learned Systems,” Expert Systems with Applications: An Int’l J., vol. 20, no. 1, 2001.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

focus

knowledge management

Postmortem: Never Leave a Project without It Andreas Birk, sd&m Torgeir Dingsøyr, Sintef Telecom and Informatics Tor Stålhane, Norwegian University of Science and Technology

n every software project, the team members gain new knowledge and experience that can benefit future projects and each member’s own professional development. Unfortunately, much of this knowledge remains unnoticed and is never shared between individuals or teams. Our experience with project postmortem analysis proves that it is an excellent method for knowledge management,1 which captures experience and

I

Although primarily used for large projects and companies, postmortem analysis also offers a quick and simple way to initiate knowledge management in small- or mediumsize software projects. 0740-7459/02/$17.00 © 2002 IEEE

improvement suggestions from completed projects and works even in small- and medium- size companies that cannot afford extensive KM investments. However, PMA has been mainly advocated for situations such as completion of large projects, learning from success, or recovering from failure.2–4 When used appropriately, PMA ensures that team members recognize and remember what they learned during a project. Individuals share their experiences with the team and communicate them to other project groups. Additionally, PMA identifies improvement opportunities and provides a means to initiate sustained change. We have applied a lightweight approach to PMA in several projects5,6 by focusing on a few vital principles: ■

■ ■

PMA should be open for participation from the entire team and other project stakeholders. Goals can—but need not—provide a focus for analysis. The PMA process comprises three phases: preparation, data collection, and analysis. For each phase, team members

can apply a number of fairly simple methods, such as the KJ method (after Japanese ethnologist Jiro Kawakita)7 that collects and structures the data from a group of people. Preparation When we conduct PMA in software companies, two software process improvement group members work as facilitators together with two to all project team members. Facilitators organize the analysis, steer the discussion, and document the results. They can be employees in the company where the PMA is conducted or external, as we are. External facilitators often have an advantage performing the PMA because participants regard them as more neutral and objective. However, they might not know the company as well as internal facilitators, so preparation is important. During the preparation phase, we walk through the project history to better understand what has happened. We review all available documents, such as the work breakdown structure, project plans, review reports, and project reports. We also determine a goal for the PMA. May/June 2002

IEEE SOFTWARE

43

Once the group identifies the important topics, we must prioritize them before proceeding with the analysis.

Goals might be “Identify major project achievements and further improvement opportunities” or “Develop recommendations for better schedule adherence.” If a PMA does not have a specific focus to guide our preparation, we briefly discuss the project with the project manager and key engineers. We find it practical to distinguish between two PMA types: One is a general PMA that collects all available experience from an activity. The other is a focused PMA for understanding and improving a project’s specific activity, such as cost estimation. It helps to explicitly state goals for both of these PMA variants during this phase. Data collection In the data collection phase, we gather the relevant project experience. Usually, project team members and stakeholders have a group discussion, or experience-gathering session. We can often conduct data collection and the subsequent analysis within the same session. You shouldn’t limit experience gathering to the project’s negative aspects, such as things to avoid in the future. Instead, maintain a balance by identifying a project’s successful aspects, such as recommended practices. For example, during a PMA at a medical software company, the team realized that the new incremental software integration process significantly improved process control and product quality. Integration had been so smooth that without the PMA, its important role might have gone unnoticed. Some techniques that we find useful for data collection include ■

■

■

44

IEEE SOFTWARE

May/June 2002

Semistructured interviews. The facilitator prepares a list of questions, such as “What characterizes the work packages that you estimated correctly?” and “Why did we get so many changes to the work in package X?” Facilitated group discussions. The facilitator leads and focuses the discussion while documenting the main results on a whiteboard. KJ sessions. The participants write down up to four positive and negative project experiences on post-it notes. Then they present their issues and put the notes on a whiteboard. The participants rearrange all notes into groups according to topic and discuss them.

Once the group identifies the important topics, we must prioritize them before proceeding with the analysis. This will ensure that we address the most significant issues first. For example, during a PMA we performed in a satellite software company, frequent and late requirements changes emerged as an important topic. A software developer commented that during the project, team members found it difficult to identify when the requirements had changed, so much so that the code had to be rewritten completely. In such situations, they made a few wrong decisions, which reduced the software’s quality. After this PMA session, other project members made requirements changes a high-priority topic for analysis. Analysis In this phase, as facilitators, we conduct a feedback session in which we ask the PMA participants: “Have we understood what you told us, and do we have all the relevant facts?” When we know that we have sufficient and reliable data, we use Ishikawa diagrams6 in a collaborative process to find the causes for positive and negative experiences. We draw an arrow on a whiteboard, which we label with an experience. Then, we add arrows with causes—which creates a diagram looking like a fishbone. In our example from the satellite software company, we found four causes for changing requirements: poor customer requirements specification, new requirements emerging during the project, little contact between the customer and software company, and the software company’s poor management of requirements documents. Because PMA participants are a project’s real experts and we have time limitations, we perform all analysis in this step. Results and experience Facilitators document the PMA results in a project experience report. The report contains ■

■

■

A project description, including products developed, development methods used, and time and effort needed The project’s main problems, with descriptions and Ishikawa diagrams to show causes The project’s main successes, with descriptions and Ishikawa diagrams

About the Authors

■

Andreas Birk is a consultant and software engineering professional at sd&m, software design and management. His special interests include software engineering methods, knowledge management, and software process improvement. He holds a Dr.-Ing. in software engineering and a Dipl-Inform. in computer science and economics from the University of Kaiserslautern, Germany. He is a member of the IEEE Computer Society, ACM, and German Computer Society. Contact him at sd&m, Industriestraße 5, D-70565 Stuttgart, Germany; [email protected].

A PMA meeting transcript as an appendix, to let readers see how the team discussed problems and successes

In an example from the satellite software company, facilitators wrote a 15-page report in which they documented the problem with changing requirements with an Ishikawa diagram that showed the four main causes. After facilitators submit a report, the knowledge management or quality department must follow up. In our experience, PMA is suitable when a project reaches a milestone and when the company is looking for qualitative experience that will help improve a similar, future project. You should not apply PMA in situations with unfinished activities, or when serious underlying conflicts might remove the focus from improvement. If the atmosphere isn’t appropriate for discussing a project’s problems, we prefer using approaches other than PMA, such as those outlined in Project Retrospectives: A Handbook for Team Reviews.2 When there have been serious conflicts in the project, this is more appropriate for managing the risk that discussions degenerate into a hunt for scapegoats. Finally, you must have enough time for following up on PMA results. In our experience, if teams apply PMA in the right setting, it is an excellent step into continuous knowledge management and improvement activities. It makes project team members share and understand one another’s perspectives, integrates individual and team learning, and illuminates hidden conflicts. It documents good practice and problems, and finally, it increases job satisfaction by giving people feedback about their work. Performing a PMA can even improve project cost estimation. We applied PMA to three projects in an Internet software development company, which all had serious cost overruns. The company could not allocate workers with skills specific to the project. This led to a need for courses—the team’s experts had to act as tutors for the rest of the team and were distracted from their roles in the project. By performing the PMA, the company realized the gravity of the qualification issue and how it led to the project going over budget. As an improvement action, a training budget was set up on the company level instead of the project level. The company no longer charged staff qualification to the pro-

Torgeir Dingsøyr is a research scientist at Sintef Telecom and Informatics research foun-

dation in Trondheim, Norway. He wrote his doctoral thesis on “Knowledge Management in Medium-Sized Software Consulting Companies” at the Department of Computer and Information Science, Norwegian University of Science and Technology. Contact him at Sintef Telecom and Informatics, SP Andersens vei 15, NO-7465 Trondheim, Norway; [email protected].

Tor Stålhane is a full professor of software engineering at the Norwegian University of

Science and Technology. He has a MSc in electronics, and a PhD in applied statistics from Norwegian University of Science and Technology. He has worked on compiler development and maintenance and software reliability, and on software process improvement and systems safety. Contact him at Department of Computer and Information Science, Norwegian University of Science and Technology, NO-7491 Trondheim, Norway; [email protected].

ject’s budget, and now views it as an investment into quality and competitive advantage. As a result of this PMA, management realized the strategic importance of staff qualification and knowledge management—a truth that often gets buried in the hectic rush of Internet software business.

W

e received a lot of positive feedback from PMA participants in different companies. Particularly, they like that PMA offers a simple yet effective way to uncover both achievements and improvement opportunities. One developer at the satellite software company noted, “If you do a PMA on the project...you have to think through things,” which is a crucial part of knowledge management. So, never leave a project without it! References 1. C. Collison and G. Parcell, Learning to Fly: Practical Lessons from One of the World’s Leading Knowledge Companies, Capstone, New York, 2001. 2. B. Collier, T. DeMarco, and P. Fearey, “A Defined Process For Project Post Mortem Review,” IEEE Software, vol. 13, no. 4, July/Aug. 1996, pp. 65–72. 3. N.L. Kerth, Project Retrospectives: A Handbook for Team Reviews, Dorset House Publishing, New York, 2001. 4. A.J. Nolan, “Learning from Success,” IEEE Software, vol. 16 no. 1, Jan./Feb. 1999, pp. 97–105. 5. T. Stålhane et al., “Post Mortem—An Assessment of Two Approaches,” Proc. European Software Process Improvement (EuroSPI 01), ICSN, Bray, Ireland. 6. T. Dingsøyr, N.B. Moe, and Ø. Nytrø, “Augmenting Experience Reports with Lightweight Postmortem Reviews,” 3rd Int’l Conf. Product Focused Software Process Improvement (Profes 01), Lecture Notes in Computer Science, vol. 2188, Springer-Verlag, Berlin, pp. 167–181. 7. D. Straker, A Toolbook for Quality Improvement and Problem Solving, Prentice Hall International, London, 1995, pp. 89–98 and 117–124.

May/June 2002

IEEE SOFTWARE

45

focus

knowledge management

Experience in Implementing a Learning Software Organization Kurt Schneider and Jan-Peter von Hunnius, DaimlerChrysler Research Center Victor R. Basili, University of Maryland

ompetence in software development and acquisition has become essential for the automotive industry. As a manufacturer of premium-class cars, DaimlerChrysler depends on high-quality software for its sophisticated electronic control units. ECUs implement features such as intelligent brake assistants, electronic stability, and engine controls. Unfortunately, software development and acquisition competencies are a scarce resource. Consequently, DaimlerChrysler decided

C DaimlerChrysler created its Software Experience Center to investigate experience reuse and encourage experience-based software process improvement. Here, the authors report on challenges the company faced when creating the SEC. 46

IEEE SOFTWARE

to better use its internal software knowledge in two ways. First, it wanted to improve software development and acquisition processes to increase software quality and repeatability of success. Second, it wanted to explicitly reuse knowledge from previous software projects to enhance future ones. In particular, DaimlerChrysler considered reusing experiences as a key to better project performance and higher software quality. We can view such experience exploitation as a variant of knowledge management.1 Unlike factual knowledge, we can’t find experience in textbooks. Experiences are related to the environment and context in which they occurred, and when reused in their original context, they can direct software process improvement (SPI). For example, we can calibrate the frequency and intensity of design or code inspections to optimize effort spent and errors detected.

May/June 2002

Here, we report on DaimlerChrysler’s Software Experience Center project. The SEC aimed to investigate experience reuse (as a variant of knowledge management) and apply insights and collected experiences to SPI. Experience-based SPI The SEC’s operational goal was to provide business units with the concepts of a learning organization and a prototype of an experience base. To do this, researchers acted as experience and knowledge engineers and coaches for the business units to assist them in their experience exploitation activities and transfer insights into SPIs. DaimlerChrysler expected a learning software organization would better use the scarce resource of available software competency.2 The SEC built on the concept of an experience factory,3 which is an organizational unit that supports several software projects. As a 0740-7459/02/$17.00 © 2002 IEEE

Learning implies broad and deep change Introducing a learning software organization deeply changes how developers and managers should work. Developers must change their mindsets, skills, and behavior; managers must change their expectations about what gets delivered and when; and the organization must reconsider its training approach and any processes that interact with the primary process it is changing. If the effects of change do not occur across the organization, improvements will not happen, and the cost and effort invested will have been wasted.

separate entity, an experience factory receives plans, status information, and experiences from all participating projects. Incoming data is organized in models, such as defect density models, Pareto charts of defect class baselines, algorithms, and so forth.4 These models provide projects with immediate feedback—such as “your error density is now 10 percent higher than usual”—and experience-based advice—“when you inspect more than five pages of code at a time, your performance goes down.” NASA first implemented this concept.5 DaimlerChrysler’s situation and environment called for some modifications.6 Its experiences were more qualitative than those of a classical experience factory because it had less quantitative data available. Consequently, it also used process models rather than parametric equations. DaimlerChrysler’s SEC supported all activities, from experience elicitation to making experience available for a software task at hand. Experience elicitation involved interviews, feedback forms, and an optimized version of a lightweight elicitation workshop.7 To spread consolidated experiences, we used an intranet description of the software process at hand (for example, of an inspection process). Using HTML, we could describe a process and link it to such items as training materials, experiences, checklists, frequently asked questions, and expert emails. We called this collection our experience base for any given topic (topics include software inspections, software risk management, and requirements engineering). An experience base is the persistent storage of experiences, organized around a software process.8 It offers email contact in different contexts and users are encouraged to provide feedback. We had some scripts that could quickly update an experience base, but beyond this, there was no automation. Elicitation and reuse procedures were more critical than tools or automation.

Experiences at DaimlerChrysler. We experienced several situations in which management commitment was limited in time and scope. For example, one manager complained about an insufficient effect of change after only two months of SEC operation. In another case, a manager constantly refocused the area for improvement. In both cases, management blocked emerging change by not understanding an important implication of change: it takes time. On the other hand, when handled with care, change can lead to success. For example, over a three-year period, a quality management support group encouraged a software risk management process by developing pilot projects, discussing the process in user groups, submitting it to management for approval and commitment, offering training in tailored courses, and using a specific, custommade risk management experience base. The group considered all stakeholders and their concerns, differences, and needs; it anticipated and addressed ripple effects (for example, the need for training).9 Risk management is now accepted in the entire department and applied to all top strategic software projects as well as to a growing number of other software projects. Risks are now identified earlier and can be mitigated more effectively.

Three major challenges for a learning software organization Establishing an SEC as a learning organization in the business units participating in this project was surprisingly difficult. We discuss two cultural challenges and one technical challenge that were decisive for turning a department or business unit into a learning software organization at DaimlerChrysler.

Recommendations. Improvement causes change, and effective change is not local. Because we cannot always control the speed of ripple effects, patience and a long-term vision are indispensable. All stakeholders must understand what is happening, why it is happening, how it affects their jobs, and why it takes so long. Things usually differ from what we first believe, so adaptation and iteration are needed. Spreading best practices in

Introducing a learning software organization deeply changes how developers and managers should work.

May/June 2002

IEEE SOFTWARE

47

Apply risk management to your software improvement initiative and learning efforts just as you would to a software development project.

experience bases can support the ripple of change and its speed. Plan activities that help stakeholders adjust to the ripple effects, and advise key people that they must adjust their own behavior—not just provide funding. Capitalizing on learning involves risk There is much to gain by making changes based on learning—for example, there are many opportunities in SPI.2 However, improvement programs and learning initiatives imply that skills must change and acquired expertise might become obsolete, which could be perceived as personal risks. Consequently, employees sometimes try to avoid the change. Ignoring or neglecting personal risks can turn into a serious risk for the entire SPI activity. Unfortunately, risk management is rarely applied to SPI, and an ignored risk is the worst kind.10 Experiences at DaimlerChrysler. We introduced a use case technique for requirements engineering in one group, not considering the risk this introduction posed for several group members. As electrical engineers, they were used to model-based development. Consequently, the new technique was neither openly rejected nor actively adopted. Because we couldn’t modify the team’s background, we decided to cancel the method introduction when we reconsidered it from a risk perspective. We thus immediately stopped wasting money and no longer risked the SPI initiative’s reputation. Instead, we refocused and concentrated on a more appropriate improvement activity (improving documentation structure). Recommendations. Apply risk management

to your software improvement initiative and learning efforts just as you would to a software development project. Use experiencebased checklists of common risks in your environment (such as personnel and budget risks), thus reusing knowledge and experience gained in earlier projects. Watch out for personal risks that an activity might create for important stakeholders. Stop an activity before you lose too much money and your reputation. Better yet, refocus activities long before risks turn into problems, and you’ll avoid failure completely. Experience value through packaging The experience base must provide infor48

IEEE SOFTWARE

May/June 2002

mation in a format that is useful for intended users. At DaimlerChrysler, qualitative experiences are most common. Packaging refers to the activity of comparing, analyzing, and combining several pieces of raw experiences received from various projects (for example, from developers or project leaders). We organize this material according to the steps of the process models. The result is a document that represents either a consolidated view of the packaged experiences or modifications to an existing (process) model. Packaging is a technical challenge because it requires identifying and working with several models.4 The key is to make experiencerelated material relevant to a real user. This includes tailoring contents and format to a concrete anticipated usage situation. Experience is only valuable when set in context. We must base iteration, evolution, and learning on explicit information to form the seed for the next cycle.11 Experiences at DaimlerChrysler. SEC researchers observed quality circles: Quality assurance staff gathered in an informal way to exchange experiences. Unfortunately, they captured little information, writing down almost nothing. The few items they did capture were hardly reusable. To assist their experience capturing, the SEC team needed to develop expertise in the subject under discussion (such as inspections and review plans, and quality management issues). Over time, the QA staff adopted this practice and wrote down more information and introduced an experience base for quality assurance. Establishing the base was a lengthy process of describing, reworking, and representing (on paper and on slides) inspection processes, support material, and review plans.8 Recommendations. Develop packaging techniques and tailor them for your organization. Domain experts should create packages based on anticipated user needs. It should be made as simple as possible to create and use packages. Be aware of the importance and timeliness of feedback. Iterate often to know what the users really need. Don’t fall in love with an idea you might have packaged, and don’t try to reach the final solution in one iteration. Let the user see, use, and comment on partial solutions over several iterations.

E

ach issue we’ve discussed was more of a challenge than tool support and automation. We have learned to better cope with these challenges, but they remain the most decisive issues in implementing a learning software organization through explicit experience exploitation. We recommend providing advice on dealing with the issues, but each organization must find its own approach. Once we recognized these challenges and dealt with them directly, the SEC ran more smoothly. It has improved many processes, and learning from experience has become a more natural part of daily life in the business units.

About the Authors Kurt Schneider is a researcher and project leader at the DaimlerChrysler Research Cen-

ter, Ulm, Germany. He currently works in software process improvement, software quality, and lightweight approaches to software development. He has led large research projects—in particular the SEC project—with several business units and international partner companies. He studied computer science at the Friedrich-Alexander Universität Erlangen-Nürnberg, Germany, and received his doctoral degree in software engineering from the Universität Stuttgart, Germany. Contact him at DaimlerChrysler Research Center Ulm, P.O. Box 2360, 89013 Ulm, Germany; [email protected]. Jan-Peter von Hunnius is a researcher and PhD student at the DaimlerChrysler Research Center, Ulm, Germany. His research interests include experience-based process improvement, software development processes in general, extreme programming, and the rational unified process. He received his Diplom-Informatiker in computer science from the Albert-Einstin Universität Ulm, Germany. Contact him at DaimlerChrysler Research and Technology, Software Process Engineering (RIC/SP), P.O. Box 2360, 89013 Ulm, Germany; jan.hunnius@ daimlerchrysler.com.

Victor R. Basili is a professor of computer science at the University of Maryland,

References 1. T.G.P. Davenport, Knowledge Management Case Book, John Wiley & Sons, New York, 2000. 2. R.V. Solingen et al., “No Improvement without Learning: Prerequisites for Learning the Relations between Process and Product Quality in Practice,” Product Focused Software Process Improvement (PROFES 2000), Springer-Verlag, New York, 2000, pp. 36–47. 3. V. Basili, G. Caldiera, and D.H. Rombach, “The Experience Factory,” Encyclopedia of Software Eng., John Wiley & Sons, New York, 1994, pp. 469–476. 4. V. Basili and F. McGarry, “The Experience Factory: How to Build and Run One,” Proc. Int’l Conf. Software Eng. (ICSE 19), ACM Press, New York, 1997, pp. 643–644. 5. V. Basili et al., “The Software Engineering Laboratory: An Operational Software Experience Factory,” 14th Int’l Conf. Software Eng. (ICSE ’92), ACM Press, New York, 1992, pp. 370–381. 6. F. Houdek and K. Schneider, “Software Experience Center: The Evolution of the Experience Factory Concept,” Int’l NASA-SEL Workshop, Proc. 24th Ann. Software Eng. Workshop, NASA Goddard Software Eng. Lab (SEL), Greenbelt, Md., 1999. 7. K. Schneider, “LIDs: A Light-Weight Approach to Experience Elicitation and Reuse,” Product Focused Software Process Improvement (PROFES 2000), SpringerVerlag, New York, 2000, pp. 407–424. 8. K. Schneider and T. Schwinn, “Maturing Experience Base Concepts at DaimlerChrysler,” Software Process Improvement and Practice, vol. 6, 2001, pp. 85–96. 9. K. Schneider, “Experience-Based Training and Learning as a Basis for Continuous SPI,” Proc. 6th Ann. European Software Eng. Process Group Conf. (EuropeanSEPG), 2001. 10. E.M. Hall, Managing Risk: Methods for Software Systems Development, Addison-Wesley, Reading, Mass., 1997. 11. G. Fischer, “Seeding, Evolutionary Growth and Reseeding: Constructing, Capturing and Evolving Knowledge in Domain-Oriented Design Environments,” Automated Software Eng., vol. 5, no. 4, Oct. 1998, pp. 447–464.

College Park, and the executive director of the Fraunhofer Center, Maryland. He is also one of the founders and principals in the Software Engineering Laboratory. He works on measuring, evaluating, and improving the software development process and product and has consulted for many organizations. He is co-editor-in-chief of the International Journal of Empirical Software Engineering, published by Kluwer. He is an IEEE and ACM Fellow.

Get access to individual IEEE Computer Society documents online. More than 67,000 articles and conference papers available! $5US per article for members $10US for nonmembers

http://computer.org/publications/dlib

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib. May/June 2002

IEEE SOFTWARE

49

focus

knowledge management

Process Knowledge Management with Traceability Balasubramaniam Ramesh, Georgia State University

nowledge management involves the collection, assimilation, and use of the explicit and tacit knowledge spread throughout an organization. (We can explicate or codify explicit knowledge; tacit knowledge cannot be codified or expressed in written form.1) In software engineering, process knowledge is any explicit or tacit knowledge about the activities, steps, and procedures involved in creating software

K Traceability helps process knowledge evolve in software development organizations. Drawing from a large empirical study, this article examines traceability’s role in facilitating critical knowledge management processes. 50

IEEE SOFTWARE

solutions. A key component in process knowledge is traceability: the ability to follow the life of an object developed during software engineering from its creation to its use. Creating and maintaining the relationships that exist among objects and people in software development helps realize traceability. This ability to produce a web of relationships is essential in the development of knowledge networks, which are formal and informal networks of people and objects and the relationships between them.2 A primary role of KM systems is to craft such networks to accumulate and use knowledge. Thus, by gluing the fragmented sources of knowledge, traceability helps create, store, retrieve, transfer, and apply process knowledge in software development organizations. Why traceability? Seamlessly connecting the knowledge

May/June 2002

fragments spread across a software development organization is a major KM challenge. For example, design rationale knowledge typically resides in different repositories or sources such as CASE tools, intranets, design notebooks, and annotation tools. The absence of any traceability to link designs to their rationale (which can contain the alternatives considered and critical design decisions or assumptions) severely restricts the usefulness of these knowledge fragments. Even design rationale authors find that interpreting the rationale they’ve created is difficult if sufficient time has elapsed since its creation. Therefore, successful knowledge transfer and reuse requires contextualized knowledge. Traceability helps achieve it by linking related knowledge fragments. The challenge of connecting knowledge fragments is more acute with tacit knowl0740-7459/02/$17.00 © 2002 IEEE

edge. Because individuals usually hold it, the ability to identify those who have the knowledge needed to solve the problem at hand is essential. Here, traceability can link objects produced in systems development with the stakeholders who played different roles in their creation, maintenance, and use. The study The observations reported in this article come from a study I co-developed of best practices in traceability over 30 system development organizations in a wide variety of industries.3 A key objective was understanding how traceability could help manage process knowledge. The participants had experience in several areas of system development including project management, system engineering, requirements management, system testing, integration, quality assurance, maintenance, and implementation. They had an average of 15.5 years of experience in systems development. The study represented a wide range of projects in terms of outlay (US$15 million to several billion dollars), size (several thousand lines to over 1 million lines), and number of requirements (600 to over 10,000). Both users (for example, system maintainers) and producers (for example, system designers) of traceability information participated. We used multiple data collection methods (focus groups, interviews, and document review) to define and confirm the findings. Traceability’s role in KM processes Maryam Alavi and Dorothy Leidner present a framework for analyzing the role of information systems in KM.4 According to this framework, we can view organizations as knowledge systems with four sets of socially enacted knowledge processes: construction, storage and retrieval, transfer, and application. In the study, we examined the best practices in traceability to identify its role in enabling each of these KM processes. Construction Construction means adding new components or replacing existing tacit and explicit knowledge components. An important aspect of knowledge construction is the transfer of knowledge from one source (individ-

ual, document, or group) to another and from one state (tacit or explicit) to another.4 Traceability helps generate explicit and tacit knowledge about software development processes and their outputs. In a telecommunications company participating in our study, we observed a comprehensive traceability scheme for maintaining design rationale for all critical design decisions. It was created within a CASE tool using document templates for various components of rationale such as assumptions and alternatives, which were linked to relevant design diagrams. An incentive scheme in which design rationale users assigned bonus points to contributors encouraged experts to explicitly document their tacit knowledge about the design process. Access to this knowledge helps less experienced designers not only learn important design tricks but also convert the explicit knowledge stored in documents into new, internalized tacit knowledge. The ability to link various related knowledge sources helps users develop new insights and ideas that were not feasible when the rationale used to appear only in standalone design notebooks. Here, traceability facilitates collaboration and communication among project teams and accelerates knowledge creation.

The ability to link various related knowledge sources helps users develop new insights and ideas that were not feasible.

Knowledge storage and retrieval Storage and retrieval are key to effectively managing a knowledge network. An aerospace company in our study realized that its designers couldn’t reuse past knowledge from related projects due to the lack of traceability between software components and their requirements. So, it built a traceability tool that linked code fragments to design elements and requirements using Corba-based object linking. The tool supports multimedia documents and provides indexing and querying facilities to trace relevant knowledge fragments stored in the knowledge base. This lets the organization readily store and access knowledge from diverse sources—something most organizations find impossible to do.5 The loss of tacit knowledge at the conclusion of individual projects and the inability to retain and reuse knowledge at an organizational level frustrated the management of a consulting organization in our study. So, it May/June 2002

IEEE SOFTWARE

51

Knowledge integration is central to its application in organizations.

tasked each project team to create an evolving knowledge map that links knowledge-intensive tasks to relevant individuals and documents in a corporate directory of knowledge sources. Many organizations use a variant such as a corporate yellow pages directory for locating both tacit and explicit knowledge. Knowledge transfer An important aspect of KM is the transfer of knowledge to where it is needed and can be used.4 An electronics manufacturer in our study fashioned a requirements management system to provide traceability between requirements, video clips of usage scenarios of its target product, and documentation on previous designs. The system also provides links to relevant experts through a corporate directory of knowledge sources. This system helps knowledge transfer by immersing designers in the context of the system they’re designing. For example, at the beginning of new engagements, a software consulting company in our study links project documentation to relevant lessons learned from similar projects stored in its knowledge base. Availability of knowledge is not sufficient for knowledge transfer. Other factors such as the receiver’s absorptive capacity and the perceived value of the knowledge are critical for successful knowledge transfer. Knowledge application Knowledge integration is central to its application in organizations. In several organizations, we observed homegrown tools that use wrappers to integrate legacy data and systems, message-brokering-based middleware, and knowledge servers for integrating various knowledge sources. An interesting example is the extension

of a collaboration support system that links design team discussions with design diagrams in Rational Rose and project documents in tools such as Microsoft Office and Project using ActiveX controls. This helps developers understand the organizational knowledge embedded in rules and routines (for example, coding and architecture standards set by the organization) and apply them appropriately. By providing appropriate contextual information, the potential for misapplying the knowledge also reduces.

T

raceability can play a crucial KM role in software organizations—if the facilities to support the KM processes discussed here are developed. Lack of well-defined traceability schemes and the inability of traceability tools to link knowledge fragments in heterogeneous environments are common problems that limit the usefulness of traceability in KM in most organizations.

References 1. I. Nonaka and H. Takeuchi, The Knowledge-Creating Company, Oxford Univ. Press, New York, 1995. 2. A. Tiwana and B. Ramesh, “Integrating Knowledge on the Web,” IEEE Internet Computing, vol. 5, no. 3, May/June 2001, pp. 32–39. 3. B. Ramesh and M. Jarke, “Toward Reference Models for Requirements Traceability,” IEEE Trans. Software Eng., vol. 37, no. 1, Jan. 2001, pp. 58–93. 4. M. Alavi and D.E. Leidner, “Knowledge Management and Knowledge Management Systems: Conceptual Foundations and Research Issues,” MIS Quarterly, vol. 25, no. 1, Mar. 2001, pp. 107–136. 5. E.W. Stein and V. Zwass, “Actualizing Organizational Memory with Information Systems,” Information Systems Research, vol. 6, no. 2, 1995, pp. 85–117.

About the Author Balasubramaniam Ramesh is an associate professor of computer information systems at Georgia State University. His research interests include requirements engineering and traceability, knowledge management, and collaborative work. Several major funding agencies have supported his work, including the US Office of Naval Research and the US National Science Foundation. He received his PhD in information systems from New York University. Contact him at 35 Broad St., Atlanta, GA 30303; [email protected]. For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

52

IEEE SOFTWARE

May/June 2002

focus

knowledge management

Knowledge Management at Infosys Shivram Ramasubramanian and Gokulakrishnan Jagadeesan, Infosys

“I was working on a proposal for a new client. The project was for a new technology, so I had to search the Internet and library for two weeks to get the details.” “You could have referred to our research division’s paper on this new technology. The details are available on our intranet!”

cenarios such as this are common across software companies worldwide. Knowledge workers alone can’t sufficiently gather and distribute knowledge. Organizations require an infrastructure to support such workers and an information flow that enables knowledge dissemination. If you look at typical software scenarios, there is a greater percentage of knowledge workers than so-called hand workers. Knowledge workers spend some time doing hand work (typing,

S Late in 2000, Infosys, a software services company, started a knowledge management initiative to better enable knowledge dissemination. The authors review this initiative, highlighting KM’s importance and benefits. 0740-7459/02/$17.00 © 2002 IEEE

writing email, and so forth), but this is just to properly format the knowledge. Hence, knowledge management assumes enormous significance. Here, we review KM practices at Infosys, a software services company headquartered in India, discussing how to implement KM and revealing some of its potential benefits. Organization-wide KM Infosys provides consultancy and software services worldwide to Fortune 500 companies. It employs approximately 10,200 employees and executes nearly 1,000 software projects at any given point in time in diverse areas such as telecom, manufacturing, insurance, finance, and so forth. Needless to say, managing knowledge at Infosys is a huge challenge.

Until late 2000, Infosys restricted KM to certain pockets within the company. Then, it began a KM initiative with a steering committee that had representation from the Board of Directors and senior management. Currently, there are approximately eight full-time people designated as brand managers who help build and maintain the KM infrastructure. A central pool created for this purpose funds the majority of the KM costs. However, for project-level KM, the software project bears the cost—typically 2 to 3 percent of the total cost. Infosys manages organization-wide knowledge using three centrally operated knowledge repositories: the Knowledge Shop (KShop), Process Asset Database, and People Knowledge Map. Infosys built the K-Shop architecture on Microsoft site server technology, and all emMay/June 2002

IEEE SOFTWARE

53

Dynamic KM is essential in a software project because in many cases the project depends on the availability of the latest knowledge.

ployees can access it through a Web interface. The company encourages people to submit papers related to technology, domain, trends, culture, project experiences, internal or external literature, and so forth. They can submit the articles in any format that the Web supports (for example, HTML or Word), and we designed templates for various content types to ensure uniformity. In addition, the K-Shop has an excellent search facility that offers search through multiple parameters. K-Shop documents are available to all Infosys employees and are segregated based on the user’s selected keywords and content type. Because only a few employees write their experiences in the form of a paper, a LAN system called the Process Asset Database captures the “as is” project deliverables. This contains project artifacts such as project plans, design documents, and test plans. Users can search the documents based on domain, technology, project type, project code, customer name, and so forth. This helps provide new projects with information on similar, previously executed projects and helps set quantitative goals. The People Knowledge Map is a directory of experts in various fields. It is an intranet-based system where employees can search and locate experts. The directory’s usability is enormous because it provides multiple nodes or topics. It serves as the bridge between two knowledge workers: the user and the provider. As mentioned earlier, we need more than one system or mechanism for KM. Infosys’s intranet portal Sparsh serves as the window for all systems and acts as the central tool. The company’s Quality System Documentation is a repository of all process-related guidelines, checklists, and templates. These serve to standardize the process followed in a project and hence the project’s outputs. Infosys also has electronic bulletin boards for dis-

Further Reading Presidents and fellows of Harvard College, Harvard Business Review on Knowledge Management, Harvard Business School Press, Boston, 1998, pp. 22–23. F. Horible, Managing Knowledge Workers, John Wiley & Sons, New York, 1999. J. Honeycutt, Knowledge Management Strategies, Microsoft Press, Redmond, Wash., 2000. J. Leibowitz, Building Organizational Intelligence: A Knowledge Management Primer, CRC Press, Boca Raton, Fla., 1999.

54

IEEE SOFTWARE

May/June 2002

cussing technical and domain-related topics. In addition, there are news groups and newsletters from various departments that discuss the latest technology and business trends. KM at the project level KM is even more important for a particular software project than it is for an entire organization. The benefits of effectively managing and sharing knowledge in a project team include the ability to ■ ■ ■

Easily react to customer requests Improve productivity through fewer defects and rework Improve teamwork

Dynamic KM is essential in a software project because in many cases the project depends on the availability of the latest knowledge. Many Infosys projects have Web sites to manage knowledge content. Each project also maintains a project-specific knowledge repository and a detailed training plan with material to tackle project attritions. In addition, projects also hold weekly knowledge sharing sessions where team members discuss and document past learning, so projects reuse knowledge effectively and can reduce effort and cost. KM in projects is everybody’s responsibility. The project manager defines the KM activities in the project plan, which then serves as a guide. Typically, about 2 to 3 percent of project effort is spent in KM activities. An incentive for knowledge sharing When a person submits a document to the K-Shop, experts chosen from the People Knowledge Map review the document in detail. If found acceptable, the K-Shop publishes it. The reviewer and author are rewarded with knowledge currency units. When an employee reads or uses a document from the K-Shop, he or she is encouraged to give KCUs for that document based on the benefits gained from reading it. Authors can accumulate KCUs for their documents and redeem them for cash or other gifts. Thus, KCUs serve twin objectives: they act as a mechanism both for rewarding knowledge sharing and rating the quality of assets in the repository. For example, suppose an author submits a document. He or she can get anywhere be-

tween three and 10 KCUs, depending on the document type. An external literature would get three points and an internal white paper would fetch 10. In cash terms, 10 KCUs translate to around .50 percent of the author’s salary. Each time another employee uses the document, the author could earn between 0 and 10 KCUs. Hence, the author’s cumulative earning will increase each time the document is used.

D

uring K-Shop’s first year, employees submitted over 7,600 documents. KM has helped Infosys increase its productivity and reduce defect levels. A rough estimate shows that Infosys reduced its defect levels by as much as 40 percent. This significantly reduces the associated rework and thus the cost of detecting and preventing defects. Also, effective reuse has increased productivity by 3 percent. All of this has been possible due to faster access to accurate information and reuse of knowledge. We’re now starting the next phase of KM

About the Authors Shivram Ramasubramanian is a software quality advisor at Infosys. His research

interests include knowledge management, metrics, and software quality models such as CMM. He graduated in mechanical engineering from Regional Engineering College, University of Calicut, and has a diploma in business finance from ICFAI, Hyderabad. Contact him at Infosys Technologies Limited, No. 138, Old Mahabalipuiram Rd., Sholinganallur, Chennai 600 119, India; [email protected].

Gokulakrishnan Jagadeesan is a software quality advisor at Infosys. His re-

search interests include knowledge management and CMM. He graduated in electrical and electronics engineering from the University of Madras and has a Masters of Engineering in advanced manufacturing technology from the University of South Australia. Contact him at Infosys Technologies Limited, No. 138, Old Mahabalipuiram Rd., Sholinganallur, Chennai 600 119, India; [email protected].

revolution, where we’ll leverage technology to further enhance its usability. We also need to separate the good content from the bad by scoring each asset. This will help us better decide which artifact to use. For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

Get CSDP Certified Announcing IEEE Computer Society's new

Certified Software Development Professional Program Doing Software Right •

Demonstrate your level of ability in relation to your peers

•

Measure your professional knowledge and competence

The CSDP Program differentiates between you and others in a field that has every kind of credential, but only one that was developed by, for, and with software engineering professionals.

"The exam is valuable to me for two reasons: One, it validates my knowledge in various areas of expertise within the software field, without regard to specific knowledge of tools or commercial products... Two, my participation, along with others, in the exam and in continuing education sends a message that software development is a professional pursuit requiring advanced education and/or experience, and all the other requirements the IEEE Computer Society has established. I also believe in living by the Software Engineering code of ethics endorsed by the Computer Society. All of this will help to improve the overall quality of the products and services we provide to our customers..." — Karen Thurston, Base Two Solutions

Register Today Visit the CSDP web site at http://computer.org/certification or contact [email protected]

May/June 2002

IEEE SOFTWARE

55

focus

knowledge management

Design and Evaluation of a Knowledge Management System Chih-Ping Wei, National Sun Yat-Sen University Paul Jen-Hwa Hu, University of Utah Hung-Huang Chen, International Semiconductor Technology

o compete in the emerging knowledge-centric economy, organizations around the world have undertaken various initiatives to manage their most important yet volatile asset—knowledge.1,2 In this article, we consider knowledge management (KM) as a systematic and organizational process for retaining, organizing, sharing, and updating knowledge critical to individual performance and organizational competitiveness.3 We can best describe our overall approach as codification-based knowledge repository building, which has become prevalent in current knowledge management practices.4

T This article discusses important implications for knowledge management in software engineering based on a system created for an integrated circuit assembly and testing firm. This system supports organization-wide knowledge creation, update, sharing, and reuse. 56

IEEE SOFTWARE

Responding to the challenge We concentrated on the design and evaluation of a KM system for an integrated circuit assembly and testing firm (see the related sidebar), where knowledge intensity, specificity, and volatility are comparable to those in software engineering and other knowledgeintense industries. Generally, such firms offer services in semiconductor product assembly and testing and therefore add value to the overall IC production and manufacturing process at a late stage. For instance, upon a wafer’s manufacture, machines automatically saw it into dies on which to perform particular functional tests. Subsequently, specialized facilities assemble and mount the dies that pass the required tests into packages. Effective KM is crucial to IC assembly and testing firms, which operate in a highly

May/June 2002

competitive industry characterized by short lead time, high-yield performance, and stringent quality standards. Typically, employees possess knowledge pertinent to the organization’s core competence, and their departure would result in substantial knowledge drainage. Additionally, knowledge retained at an individual level is difficult to pass on and reuse throughout the organization. Anchoring in a widely adopted knowledge process,5–7 we responded to the KM challenge by developing a system that supports organization-wide knowledge creation, update, sharing, and reuse. Based on our findings, we also discuss important implications for KM in software engineering, where knowledge sharing and reuse are equally critical to ultimate design quality, organization productivity, and service level.8,9 0740-7459/02/$17.00 © 2002 IEEE

Design and implementation Basically, our system architecture comprises a lessons-learned knowledge repository, a case repository, an organizational directory, a communication and distribution layer, and a set of common knowledge-management functions. Consistent with existing practice, we structured the knowledge in the system as cases rather than embedding it in the product design or quality review process. The lessons-learned knowledge repository contains validated knowledge, whereas the case repository stores previously disapproved cases or those undergoing validation. A case in either repository includes a problem or phenomenon description, analyses, comments, and recommendations for treatment, as well as the contributors’ contact information. The organizational directory contains organization structure, individuals’ profiles and contact information, and the names of domain experts in various areas. The system also has a communication and distribution layer integrated with the existing Lotus Notes email and groupware systems. This layer is vital to knowledge creation and update and offers users convenient access to both repositories. The system supports a set of common KM functions—knowledge creation, maintenance, search, and usage analysis, for example. We implemented the system using Notes Script running on Windows NT Server 4.0 (with Service Pack 6.0). We built the repositories, organizational directory, and access log using Notes Domino 4.62. Additionally, we developed the client browser and email services using Lotus Notes 4.62. Typical knowledge creation begins when an individual posts a problem. The relevant section and department managers then assess the problem’s significance. A case considered to be insufficiently significant is stored in the case repository, rather than undergoing knowledge solicitation and validation. When approving a case, a department manager identifies and selects experts to give analyses, comments, or recommendations. Notified by email, these individuals collaborate to analyze and solve the problem using the groupware system. The participating experts can access all analyses and comments submitted to the system, which fosters synergetic behavior. The department manager monitors the intensity of case discussion—

measured by the number of submitted comments—and, when it levels off, concludes with the final significance assessment and knowledge validation. The manager then stores the validated and approved knowledge in the lessons-learned knowledge repository, which is indexed by product type, manufacturing process stage, concerned department, and so on. Additionally, the KM system automatically “pushes” the newly created knowledge via email to the potential users, the problem poster, and the solicited experts. Knowledge maintenance follows a similar process, which is activated by an approved knowledge update request and completed with the dissemination of updated knowledge to potential users.

Results suggested that the system’s use significantly facilitated individual knowledge sharing.

System status and evaluation The system became operational in April 2001. Immediately after the system was available, the firm’s IS department provided user training throughout the organization. As of December 2001, employees had created a total of 1,204 cases. Production, research and development, process engineering, equipment engineering, and quality assurance departments were the most active in knowledge creation and update, and system use. The turnaround time for knowledge creation was approximately five days, a significant improvement over the previous manual process. Our evaluation compared knowledge sharing and subsequent improvements in productivity and customer service before and after the system’s availability. Seventy-eight employees—including managers, engineers, and support staff from the previously-mentioned knowledge-intensive departments— participated in the evaluation. A majority of the respondents held nonmanagerial positions (85 percent), and many of them had occupied their current posts for more than one year (48 percent). We measured each evaluation dimension using multiple question items based on a five-point Likert scale, with one being “extremely disagree” or “extremely frequent” and five being “extremely agree” or “extremely infrequent.” Our comparative analysis results suggested that the system’s use significantly facilitated individual knowledge sharing in terms of the ease of locating experts for target knowledge and sharing personal knowlMay/June 2002

IEEE SOFTWARE

57

Study Firm Overview Located in southern Taiwan, the study firm specializes in total solutions to tape-carrier-package and integrated circuit smart-card assembly and testing. Its sales revenues have grown rapidly, reaching US$7 million in 2000 (a 700 percent increase since its launch in 1998). The firm has a fairly sophisticated technology infrastructure, which consists of several integrated information systems for transaction processing, management control, and decision support. Managers at all levels value knowledge sharing and collaborative problem solving. The firm created several centralized repositories early on to provide convenient access to its structured internal knowledge, such as standard operating procedures, process guidelines, and organization policies. The firm emphasized individual continued learning of task-specific knowledge through interdepartmental product reviews or case discussions, for example. However, the firm often maintained knowledge generated by these collaborative efforts in paper-based documents. Searching for or accessing this knowledge was difficult and time-consuming. One of the firm’s central challenges had been acquiring, organizing, sharing, and using important lessons learned from previous design reviews and customer service. In response, we developed a system that retains and makes available such core knowledge through collaborative problem solving. We felt that the study firm, when equipped with the system, would improve product design at early stages and enhance its customer relationship management. We integrated our system with the existing technology infrastructure, which was familiar to the target users, including managers, engineers, and support staff.

edge with colleagues (significant at the 95 percent level). The system’s use also enhanced productivity. Particularly, the subjects’ responses suggested that using the system reduced the frequency of production interruptions due to a lack of knowledge support or access, or prolonged trouble shooting for equipment malfunction (significant at the 95 percent level). Additionally, use of the system significantly reduced the frequency of delayed replies to customer inquiries, partly because of a considerable reduction in the reanalysis or re-solving of previously examined problems. Implications for KM in software engineering Difficulties such as poorly defined requirements, frequent staff turnover, and volatile hardware and software platforms constantly challenge software engineering projects.10 These challenges require a casebased approach to organizational learning of important lessons from previous experiences or emerging requirements.11 By encompassing a fundamental KM process underpinned by effective learning from previous designs and practices, a software engineering organization can improve its 58

IEEE SOFTWARE

May/June 2002

project planning, implementation, and control. As illustrated in the case reported, implementing a validated and accessible organizational repository that contains knowledge from various projects can support and enhance such KM processes. In addition to highlighting a promising use of KM in IC assembly and testing, this study also has several implications for KM in software engineering. Specifically, management support, integration with existing technology infrastructure, and an organizational culture that values knowledge creation and sharing are critical success factors for KM system implementation. Integrating a KM system with existing technology is particularly important, because it will reduce individuals’ learning time and cognitive load and foster system acceptance and use. A system’s ease of use is relevant and might greatly depend on its interfaces with users. Findings from interviews with engineers and support staff suggested the need for further streamlining of problem-posting procedures and for concise presentation of problems, analyses, and recommendations. When supported by a KM system that institutionalizes and supports sharing critical, task-specific knowledge previously retained by individuals, software engineers and developers, like their IC assembly-testing counterparts, should be able to design and deliver better systems at a faster pace.

S

everal research directions are worth continued investigation. For instance, we have yet to examine the system’s cost-effectiveness. This study demonstrates the system’s positive effects on knowledge sharing, productivity, and service level but does not address costs. Also interesting is the development and evaluation of similar systems in the software engineering context, where project processes and stages are usually dynamic and intertwined, and knowledge validity is profoundly important. References 1. D. Leonard-Barton, Wellsprings of Knowledge: Building and Sustaining the Sources of Innovation, Harvard Business School Press, Boston, 1995. 2. G.V. Krogh, K. Ichijo, and I. Nonaka, Enabling Knowledge Creation: How to Unlock the Mystery of Tacit Knowledge and Release the Power of Innovation, Oxford Univ. Press, Oxford, UK, 2000.

3. T.H. Davenport, D.W. DeLong, and M.C. Beers, “Successful Knowledge Management Projects,” Sloan Management Rev., vol. 39, no. 2, Winter 1998, pp. 43–57. 4. M.T. Hansen, N. Nohria, and T. Tierney, “What’s Your Strategy for Managing Knowledge?” Harvard Business Rev., vol. 77, no. 2, Mar./Apr. 1999, pp. 106–116. 5. S. Staab et al., “Knowledge Process and Ontologies,” IEEE Intelligent Systems, vol. 16, no. 1, Jan./Feb. 2001, pp. 26–34. 6. V. Grover and T. H. Davenport, “General Perspectives on Knowledge Management: Fostering a Research Agenda,” J. Management Information Systems, vol. 18, no. 1, Summer 2001, pp. 5–21. 7. G. Fischer and J. Ostwald, “Knowledge Management: Problems, Promises, Realities, and Challenges,” IEEE Intelligent Systems, vol. 16, no. 1, Jan./Feb. 2001, pp. 60–72. 8. T.C. Lethbridge, “What Knowledge Is Important to a Software Professional?” Computer, May 2000, vol. 33, no. 5, pp. 44–50. 9. D.E. O’Leary, “How Knowledge Reuse Informs Effective System Design and Implementation,” IEEE Intelligent Systems, vol. 16, no. 1, Jan./Feb. 2001, pp. 44–49. 10. T.K. Abdel-Hamid, “The Slippery Path to Productivity Improvement,” IEEE Software, vol. 13, no. 4, July/Aug. 1996, pp. 43–52. 11. S. Henninger, “Tools Supporting the Creation and Evolution of Software Development Knowledge,” Proc. 12th Int’l. Conf. Automated Software Eng., IEEE CS Press, Los Alamitos, Calif., 1997, pp. 46–53.

About the Authors Chih-Ping Wei is an associate professor in the Department of Information Management at National Sun Yat-Sen University. His research interests include knowledge discovery and data mining, information retrieval and text mining, knowledge management, multidatabase management and integration, and data warehouse design. He received his BS in management science from the National Chiao-Tung University and his MS and PhD in management information systems from the University of Arizona. He is a member of the ACM and IEEE. Contact him at the Department of Information Management, National Sun Yat-Sen University, Kaohsiung, Taiwan; [email protected]. Paul J. Hu is an assistant professor of information systems for the David Eccles School of

Business at the University of Utah. His research interests include information technology management and applications in health care, electronic commerce, human-computer interaction, knowledge management, and information systems project management. He received his PhD in management information systems from the University of Arizona. Contact him at Accounting and Information Systems, David Eccles School of Business, 1645 E. Campus Center Drive, 108 KDGB, Univ. of Utah, Salt Lake City, UT 84112; [email protected]. Hung-Huang Chen is a senior MIS manager at Interna-

tional Semiconductor Technology. His research interests include knowledge management and data mining. He received a BS in mechanical engineering from Tamkang University and an MBA in information management from National Sun Yat-Sen University. Contact him at [email protected].

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

Get thousands of dollars worth of online training— FREE for members

Choose from 100 courses at the Subjects covered include… * Java * PowerPoint * Cisco * Windows Network Security With * * *

Ne Mem w 2002 be Ben rship efit

IEEE Computer Society’s Distance Learning Campus. * * * *

Project management Visual C++ TCP/IP protocols Unix

* HTML * Visual Basic * CompTIA

this benefit, offered exclusively to members, you get… Access from anywhere at any time * Vendor-certified courseware A multimedia environment for optimal learning * A personalized “campus” Courses powered by KnowledgeNet®—a leader in online training

Sign up and start learning now! http://computer.org/DistanceLearning

May/June 2002

IEEE SOFTWARE

59

focus

knowledge management

Toward a Practical Solution for Capturing Knowledge for Software Projects Seija Komi-Sirviö and Annukka Mäntyniemi, VTT Electronics Veikko Seppänen, University of Oulu

arely has a professional field evolved as quickly as software development. Software organizations are continuously struggling to keep abreast of new technologies frequently changing customer requirements; and increasingly complex software architectures, methods, and tools. Recently, many organizations have come to understand that to succeed in the future, they must manage and use knowledge more effectively at individual, team, and organizational levels.1,2 Efficient creation, distribution, and reuse of up-to-date knowledge are critical success factors that unfortunately remain difficult to achieve in practice.3–5

R

A needs-based approach to reusing software developmentrelated knowledge can overcome past failures at knowledge reuse. 60

IEEE SOFTWARE

In the context of software engineering, we define knowledge management as a set of activities, techniques, and tools supporting the creation and transfer of SE knowledge throughout the organization. One use of KM is to support software process improvement (SPI) activities. This support is important because both software engineering and quality management techniques fail if they are not based on a thorough knowledge of what is needed and what has been done in a software development organization. So, how can the existing knowledge in a software organization be captured efficiently? To try to answer this question, we conducted a case study in an independent business unit of a global corporation developing softwareintensive electronic products. The company wanted to improve the capture and reuse of

May/June 2002

software development knowledge for a particular project. It had made several attempts to improve knowledge reuse, but all these attempts had failed. Why did the earlier attempts not succeed? What would be a working solution? We set out to study those questions to learn from the previous difficulties and build on the previous successes. Analyzing the status of KM-based SPI Employee interviews and relevant documents revealed that both the managers and designers felt that a lot of knowledge was being wasted. Existing knowledge was difficult to find, and when found it was not reusable. The practices implemented earlier had obviously not been successful. The underlying goal had been to reduce software defects by increasing the knowledge 0740-7459/02/$17.00 © 2002 IEEE

transfer between different projects. The information to be shared was stored in a Lessons to Learn database. Interviews clearly indicated that many project managers were not aware of the database and few used it. An analysis of the database revealed that a number of entries were incomplete and only one of the four thematic sections was in active use. According to the database concept owner, this was because preparing database entries was time-consuming and administering the data was difficult. Moreover, the data’s accuracy and relevancy were not obvious, because most of the data was provided without structure. Another way to share knowledge between projects was Data Transfer Days. These meetings focused on analyzing past problems and success stories. The participants captured and shared important knowledge during the meetings, even though they had trouble remembering past successes and pitfalls once their projects had ended. The intention was to analyze, package, and save the results of these meetings as a reference for new projects. Unfortunately, the enthusiasm usually disappeared at this point. The meetings were useful mainly for those who were able to attend. Nevertheless, free face-to-face conversation between group members turned out to be a better way of sharing knowledge than the database (compare this to Thomas Davenport and Laurence Prusak’s idea that the human network is a highly efficient knowledge-sharing mechanism1). As you can see, neither Data Transfer Days nor, particularly, the Lessons to Learn Database were working as initially intended. An obvious reason for the latter was that the project management processes did not incorporate guidelines for the storing or searching of knowledge. Efficient use of the database would have required more disciplined processes and much more effort at capturing, packaging, searching, maintaining, and reusing the knowledge. Furthermore, most project managers were too busy coping with their everyday problems and were unwilling to undertake any further duties. Our interviews also indicated that software designers tended to trust anyone nearby, rather than any specific experts or the shared database. Researchers who have investigated these problems elsewhere have found problems similar to ours and give these reasons for reuse failures: the knowledge capturing

process is too informal, is not incorporated into the engineering processes, or is not supported by the structures of the organization.6 Davenport and Prusak have stated that if you start with technology-centered solutions (for example, a database) and ignore behavioral, cultural, and organizational change, the expected advantages never materialize.1 Looking for a new solution The organization set a challenging requirement: new solutions should have minimal impact on the software development organization and processes and should not require new technologies. Because the existing processes should not be touched, simple, manual, and offline means were preferable, removing the excessive burden of KM. This SPI action aimed to create a process that would help the company acquire experience from existing sources—such as the company’s databases and individuals—that it could apply to ongoing SE projects. One new idea to define and test was to use the SPI experts as knowledge-capturing agents instead of having software developers do it by themselves, on the fly. This method viewed projects as individual customers that required specific knowledge. Efforts would focus on the customer’s current knowledge requirements, as opposed to a large-scale acquisition, analysis, packaging, sharing, and updating of knowledge for subsequent projects. The company’s SPI persons and external experts established a new approach for capturing knowledge. This new approach consisted of a knowledge-capturing project and customer projects. The former gathered knowledge from relevant sources and packaged and provided it to a customer project for reuse on demand. This solution neither changed the organizational setting nor required any new tools. The knowledge would come from existing sources such as project final reports, error databases, discussion forums, and—most important—people. Later, this assumption proved true. Figure 1 shows a simplified capturing process. Unlike other approaches,6,7 this one did not expect ongoing software projects to supply their experience. The knowledge-capturing project is similar to the analysis organization in the Experience Factory framework6 in that it analyzes knowledge and packages it into reusable assets. However, it does this for the

The underlying goal had been to reduce software defects by increasing the knowledge transfer between different projects.

May/June 2002

IEEE SOFTWARE

61

Figure 1. The knowledgecapturing process.

1. Define scope and requirements for knowledge capturing

2. Acquire knowledge

3. Package knowledge Knowledge package

Need

customer projects’ immediate needs. Together with the company, we tested this approach on a project that urgently needed interface-related knowledge. This special knowledge was spread among various documents, memos, databases, and people. The Lessons to Learn Database did not provide this knowledge; Data Transfer Days hadn’t helped. The customer project’s needs were structured to indicate what specific knowledge was required, what form of knowledge was needed, and how the knowledge would be reused. The needs were also divided into process- and product-related knowledge. The former included, for example, software design and testing tasks, roles, organizations, skills, methods, and tools. The latter included descriptions and interfaces of products and product family hierarchies. As planned, we followed the knowledgecapturing process, using semi-structured interviews as the main technique to acquire knowledge. The process took 300 hours; the most laborious phase was experience packaging, which took more than one-half of the

About the Authors Seija Komi-Sirviö is a research scientist at the Fraunhofer Center for Experimental Software Engineering, Maryland. She is visiting from VTT Electronics, Finland, where she has carried out research into software process improvement and metrics in applied research projects from 1994 onwards. As a research group manager, she has been responsible for initiating and managing both applied research projects and industrial development projects for a broad range of clients in software engineering. Her current research interests include software process and product improvement, measurement, and knowledge management. She received her MSc in information processing science from the University of Oulu, Finland. She is a member of the IEEE Computer Society. Contact her at [email protected]. Annukka Mäntyniemi is a research scientist at VTT Electronics, Oulu, where she has worked since 1998. She received her Master’s degree in Information Processing Science from University of Oulu, Finland in 2001. Her thesis concerned the reuse of software development experiences. Her current research interests involve utilizing knowledge management in software process improvement and software reuse. Contact at [email protected].

Veikko Seppänen is a software business research professor at the University of Oulu,

Finland with almost 20 year’s experience in software research and development. He finished his engineering doctoral thesis on software reuse in 1990 and his second dissertation on economic sciences in 2000. Seppänen has published about a hundred scientific and practical publications. He was an Asla Fulbright scholar at UC Irvine in 1986-87 and a JSPS Postdoctoral Fellow at Kyoto University in 1991-93. His present research involves software business strategies and models, including value network based approaches to industrial marketing, acquisition and use of commercial-off-the-shelf software components and products, and knowledge-driven software product and business development methods. Contact him at [email protected].

62

IEEE SOFTWARE

May/June 2002

time. The delivered interface knowledge package fully met all its requirements: the customer project retrieved needed knowledge of the existing interfaces. The selected approach worked well, and the customer project was served the required knowledge just in time.

A

lthough we acknowledge the limitations of a single case study, we do not hesitate to call into question technology-centered solutions as the main means for managing software development knowledge. We feel our study is a first step toward a more comprehensive needs-based KM approach. We will continue to expand the use of the just in time KM process and will work to make it a part of normal project initiation procedures. Over time our efforts should help provide structured and packaged information that will have real value for software projects.

References 1. T.H. Davenport and L. Prusak, Working Knowledge: How Organizations Manage What They Know, Harvard College Business School Press, Boston, 1998, p. 199. 2. I. Nonaka and H. Takeuchi, The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation, Oxford Univ. Press, New York, 1995, p. 284. 3. V. Basili et al., “Implementing the Experience Factory Concepts as a Set of Experience Bases,” Proc. 13th Int’l Conf. Software Eng. and Knowledge Eng. (SEKE 01), Knowledge Systems Inst., Skokie, Ill., 2001, pp. 102–109. 4. T. Kucza et al., “Utilizing Knowledge Management in Software Process Improvement: The Creation of a Knowledge Management Process Model,” Proc. 7th Int’l Conf. Concurrent Enterprising (ICE 2001), Univ. of Nottingham, Center for Concurrent Enterprising, Nottingham, UK, 2001, pp. 241–249. 5. K. Schneider, “Experience Magnets: Attracting Experiences, Not Just Storing Them,” Proc. 3rd Int’l Conf. Product Focused Software Process Improvement (PROFES 2001), Lecture Notes in Computer Science, no. 2188, Springer-Verlag, Heidelberg, Germany, 2001, pp. 126–140. 6. V. Basili, “The Experience Factory,” Encyclopedia of Software Eng., vol. 1, John Wiley & Sons, New York, 1994, pp. 469–476. 7. A. Birk and C. Tauz, “Knowledge Management of Software Engineering: Lessons Learned,” Proc. 10th Conf. Software Eng. and Knowledge Eng. (SEKE 98), Knowledge Systems Inst., Skokie, Ill., 1998, pp. 24–31.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

feature usability

Pleasure and Pain: Perceptual Bias and Its Implications for Software Engineering Sheila Guilford, Gordon Rugg, and Niall Scott, University College Northampton

W Pain research indicates a bias in human perception that has implications for software development. The authors discuss how they investigated this bias and share their findings. 0740-7459/02/$17.00 © 2002 IEEE

hat does the journal Pain have to do with software design, debugging, and evaluation? At first glance, not a lot. However, a paper published in that journal reported a seemingly unlikely effect related to patients’ perception of unpleasant medical procedures.1

In essence, the study found that patients’ rating of overall pain during a painful medical examination was closely tied to two particular moments: the point during the procedure when the pain was most intense, and their experience in the procedure’s final moments, regardless of pain levels. Given this, patients’ perception of pain might be significantly different than their actual experience of pain, particularly if the final moments are relatively painless. How might such findings relate to software usability? We investigated this question through a series of experiments. Here, we describe our studies, their results, and the implications they hold for software engineering. Usability and pain The software industry generally views usability as a major system design and development issue, and usability researchers have drawn explicitly on a considerable body of previous psychological research. Specifically, the shift toward user-centered design has increased researchers’ use of approaches

such as user modeling and feedback. However, even evaluation approaches that don’t involve direct feedback from users—such as cognitive walkthroughs2 and heuristic evaluation3—focus on the software’s usability, rather than on its functionality. One of Rolf Molich and Jakob Nielsen’s evaluation heuristics,3 for example, is that text should be written in simple and natural language, which is important to users but irrelevant to functional issues. Usability literature also strongly emphasizes making software pleasant for users, removing features that might confuse or irritate them. Despite this emphasis on enhancing usability and reducing irritation, usability literature has paid surprisingly little attention to pain research. There is considerable literature on this topic that highlights how people’s perceptions of unpleasant experiences differ markedly from traditional expectations. (Such differences extend even to the definition of “pain,” which the literature hotly debates.) Motivated by the fact that poorly designed software can create inMay/June 2002

IEEE SOFTWARE

63

Table 1 Results from first experiment Subject

Peak

1 2 3 4 5 6 7 8 9 10 Mean

7 11.5 11 12 13 11.5 10.5 20 13.5 15 12.5

Final

1.50 8.50 9.00 10.50 11.00 10.00 5.00 15.50 7.00 6.50 8.45

Peak–end

Global

4.25 10.00 10.00 11.25 12.00 10.75 8.00 17.75 10.25 10.75 10.75

10.00 3.00 11.00 14.50 14.50 11.00 3.00 14.50 14.00 12.00 10.50

Mean

2.25 8.30 8.70 8.90 10.30 10.40 6.80 14.60 8.90 11.15 9.03

Predictor

Peak–end Mean Peak–end Peak–end Peak–end Peak–end Mean Mean Peak–end Mean Peak–end

tensely annoying and unpleasant user experiences, Niall Scott wrote a postgraduate thesis examining how we might apply pain research findings to usability issues.4 Scott’s focus was on the “peak and end effect,”1 an idea originating in judgment and decision-making literature.5 Donald Redelmeier and Daniel Kahneman discovered the effect during a study in which they first asked patients undergoing a painful medical examination to rate their pain at regular intervals during the procedure. Then, following the procedure, they asked patients to assign their pain an overall rating retrospectively. Some people might expect these retrospective ratings to be close to the procedural ratings’ arithmetic mean; a more psychologically sophisticated prediction would favor a primacy and recency effect, in which patients’ first and last ratings would best predict the global retrospective rating. However, neither proved true in the medical study. The retrospective ratings—and, correspondingly, patients’ memory of the overall pain levels—were best predicted by a combination of the worst pain patients experienced during the procedure (regardless of when it occurred) and the pain they experienced in the procedure’s final minutes. This finding is consistent with those from human memory literature (see, for example, studies by Alan Baddeley6). Redelmeier and Kahneman’s study had significant ethical and practical implications for medicine, which Kahneman and collaborators investigated in subsequent research. One implication that they confirmed, for example, was that if at the end of painful examinations, doctors added some medically trivial, low-pain examinations, patients would remember the overall session 64

IEEE SOFTWARE

May/June 2002

as being less painful, even though the session’s total quantity of pain increased. Experiments with the peak–end effect Using Scott’s thesis as a foundation, we conducted subsequent experiments to test the applicability of the peak–end effect to software engineering. Given the domain differences, it was impossible to simply replicate Kahneman and Redelmeier’s exact experiments, and our findings are indicative rather than conclusive. Scott’s original experiment In his original work, Scott attempted to replicate the Kahneman and Redelmeier experiment as closely as possible. He also analyzed qualitative differences between the pain domain and that of usability evaluation. Scott’s subjects were 10 students using a software package as part of their course. He studied the subjects while they used the software, and asked them at regular intervals to rate their feelings about the software. Users recorded their ratings on paper, using a 20point Likert-style scale ranging from 0 (“content”) to 20 (“unhappy”). At the session’s end, Scott asked users to rate their overall feelings on a scale of –10 (“pleasant”) to 10 (“unpleasant”). He used a minus number scale for global ratings to reduce the risk of subjects simply copying their response from the last time-sliced rating. Table 1 shows the results; for clarity, we converted the global ratings to a 0–20 scale. In this table and those that follow, lower ratings indicate higher pleasure, and ■ ■ ■ ■ ■ ■

Peak indicates users’ highest rating at any point during the task Final indicates users’ last rating during the task Mean indicates users’ average rating during the task Global indicates users’ overall rating immediately after task completion Peak–end indicates the average of users’ peak and final ratings Predictor indicates the best predictor of the global rating (peak–end or mean)

A clear finding that emerged from subjects’ comments during the experiment was that having a researcher interrupt them at

Table 2 Results from second experiment regular intervals to gather data was intrusive. This was not surprising. Redelmeier and Kahneman’s subjects were not active participants in the task; having a researcher present might well have provided a welcomed distraction from an unpleasant and prolonged medical procedure. Scott’s subjects, on the other hand, were being interrupted while trying to achieve various goals. Nonetheless, the data does show a peak–end effect, although as a tendency rather than an invariable rule. For six out of the 10 subjects, the peak–end value was a better predictor of the global value than the mean rating. This is an interesting outcome because it differs from expectations of both naive models and those that are more psychologically sophisticated, such as primacy and recency. Although recency is involved here, primacy is not. Anecdotally, upon hearing the findings, several colleagues claimed with hindsight that there was nothing surprising about them. However, no one predicted this effect with foresight (another instance, presumably, of hindsight having 20/20 vision). Follow-up experiments Our obvious next step would be to see whether we could collect data less intrusively. We considered an apparently simple method, using a software tool to gather data while subjects used the software. Although the approach has obvious advantages, and subsequent researchers have used it and found evidence of the peak–end effect,7 the less obvious disadvantages (discussed in “Outstanding issues” below) led us to investigate other approaches. We also had questions about the cases in which the peak–end value was not the best predictor. The peak–end effect is based on a human cognition model that strongly predicts biases in human information processing. Using this model, we expected the effect to be a strong one, perhaps even the norm. Why, then, were there several subjects who did not behave as the model predicted? To investigate this, we decided to replicate Scott’s study on a larger scale with two different user groups. Our goal was to see whether the manual approach could reliably detect an effect despite its intrusiveness. For comparability across studies, we used the same experimental design. We

Subject

Peak

1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mean

130 83 18 30 7 0 29 47 25 28 27 15 59 63

Final

150 150 6 40 0 0 37 11 103 100 37 70 139 108 68

Peak–end

Global

Mean

Predictor

140.0 116.5 12.0 35.0 3.5 0.0 33.0 29.0 64.0 64.0 32.0 42.5 99.0 85.5

138 150 44 59 30 7 52 30 44 59 44 51 110 51 62.07

143.4 127.9 82.5 60.0 49.3 8.1 36.4 59.5 61.0 110.0 40.9 60.6 115.6 100.0 75.37

Peak–end Mean Peak–end Mean Mean Mean Peak–end Peak–end Mean Peak–end Mean Peak–end Mean Mean

again collected data at regular intervals but with more sensitive data collection using Likert-style 150-mm visual scales, anchored at one end with “pleasant” (0) and at the other with “unpleasant” (150). We used the same scale for the global value. Our subjects were students from two different courses, using a software package as part of their course. Experiment 2 had 14 subjects, and Experiment 3 had 11. We gathered data from each group separately and in the same manner as in Experiment 1. Our initial analysis uncovered something interesting: When we calculated the mean of the highest session rating and the final rating, we found the peak–end effect only in a minority of cases—five of the 25 total subjects in the two experiments. The implication was that users’ most unpleasant experience with the software (their highest rating) was not a particularly strong predictor of their retrospective feelings about it. We then decided to look at pleasure rather than pain as a predictor. We found that calculating the mean of the lowest rating (the most pleasant experience) and the final rating produced a peak–end effect in about half the cases across the two groups (11 out of the 25 subjects). Tables 2 and 3 show the results from Experiments 2 and 3, respectively. In both cases, we calculated the peak–end rating using subjects’ lowest value (most pleasant experience). Our findings suggest that in the second and third experiments, good experiences outweighed bad ones as predictors of global retrospective ratings. However, when we reanalyzed Scott’s data from Experiment 1 using this approach, the prediction rate of the May/June 2002

IEEE SOFTWARE

65

Table 3 Results from third experiment Subject

1 2 3 4 5 6 7 8 9 10 11 Mean

Peak

Final

34 25 15 40 22 34 27 11 31 15 38

122 20 53 34 7 33 55 7 97 15 46 44

Peak–end

78.0 22.5 34.0 37.0 14.5 33.5 41.0 9.0 64.0 15.0 42.0

Global

Mean

Predictor

33 59 44 37 30 29 37 29 51 44 66 41.73

78.25 59.00 47.75 70.38 41.88 39.00 82.50 31.63 83.63 63.63 63.00 60.06

Peak–end Mean Mean Peak–end Mean Peak–end Peak–end Mean Peak–end Mean Mean

peak–end effect dropped from six out of 10 subjects to three out of 10. Quite what this shows is an interesting question, and one that requires further investigation. Possible explanations would be that the decline in peak–end effect was a result of our change from a numeric to a visual scale or our change from Scott’s original “negative to positive” scale to our “zero to 150” scale for global ratings. Outstanding issues The peak–end effect appears in a significant proportion of users’ software evaluations. Depending on how we define it, the effect was evident for 20 to 50 percent of the subjects in our study. However, the details of the effect’s influence on user perceptions are still far from clear, and there are several outstanding issues that require further investigation. Domain differences Differences between the medical and software domains are complex and have farreaching implications for research methodology. Such implications must be resolved before researchers can carry out large-scale work in this area. A key difference between our domain and that of Redelmeier and Kahneman’s medical experiments was that subjects in the latter were unlikely to experience any significant pleasure during the procedures. Software users, however, can experience pleasure, and the ratings clearly show that our subjects did. Researchers continue to debate the respective natures of pleasure and pain: Are they, for example, opposite ends of a single scale, or two separate things? The complexity of such issues makes it difficult to directly extrapolate from the medical to the software domain without re66

IEEE SOFTWARE

May/June 2002

search to test the extrapolations. Another important domain difference is that the medical study involved a single continuous process that had a clear beginning and end. Participants can thus easily reflect on the overall experience. Software use, in contrast, usually involves several discontinuous activities, each of which has its own beginning and end. For example, users might pull down a menu, choose an item, work from another window, then close that window and return to the previous window. Some of these activities are very brief but can be annoying—for example, when a word processing package decides you need help and pops up an intrusive help feature. Participants might experience both painful and pleasant activities within the experiment, which they might then find difficult to globalize. In our experiment, such events were limited because it was based on time sampling, rather than being event-based. Further experimentation is needed using both a larger sample size and, ideally, a smart tool that can log details of the activity while recording the user’s satisfaction rating. Data gathering Although automated data collection is less intrusive, it is not necessarily the best choice for measuring user satisfaction with software. In addition to missing smaller annoyances (such as with the pop-up help example), automated tools cannot necessarily tie a user rating to a specific event. Knowing that a user’s peak value was indicated, say, 10 minutes into the session does software developers little good unless they know which task the user was engaged in at the time. There are thus significant advantages to using a human observer, despite the fact that their intrusiveness seems to mask the peak–end effect. At the most obvious level, human observers ask users to rate their feelings when they are clearly experiencing pleasure or pain. The drawback, of course, is that users wrangling with recalcitrant software are unlikely to welcome the request to share their feelings at such a moment. Fortunately, the literature on judgment and decision-making offers a neat (and again, counterintuitive) solution to the problem. Evidence strongly suggests that outside observers can assess a subject’s emotion with surprising accuracy.1,8 Given this, we envision future experiments

Judgment and Decision-Making Literature Although the judgment and decision-making literature has significant implications for software engineering, the implications have received comparatively little attention. The field’s seminal text is the 1982 book, Judgement under Uncertainty: Heuristics and Biases.1 The text contains classic articles that show numerous shortcomings in human judgement and decision-making. A particularly interesting feature of these shortcomings is that a high proportion of them affect experts just as much as novices, suggesting that underlying features of human cognition are involved, rather than simple ignorance. Two decades of subsequent research have confirmed the robustness of these findings in a range of real-world contexts with real-world implications, including the pain research that is the basis for our article.2–5 A full description of judgment and decision-making shortcomings and their implications would require an article of its own, but there are two key themes running through this research that are particularly relevant to software engineering: ■ ■

The vast majority of people, including experts, have significant conceptual problems in dealing with probability estimation. People are very bad in general at predicting their own future actions.

Although the robustness of these findings is well established, there is debate within the judgement and decision-making community about the extent to which they reflect real shortcomings in human cognition. Gerd Gigerenzer, for example, argues that although humans are bad at dealing with problems involving probability estimation, they are good at handling the same problems if they are rephrased as estimates of frequency.6 Similarly, other researchers argue that many of the alleged shortcomings are actually sensible heuristics for dealing with real-life situations where there are no algorithmic best solutions. An excellent overview of the topic is provided in the book Subjective Probability, which includes contributions from leading researchers in this area.7 Despite these debates, there is consensus within the judgement and decision-making field that significant and predictable shortcomings in human cognition do occur. This has meaningful implications for many areas of software engineering, including risk estimation, and forecasting and evaluating user satisfaction. We are working on several related projects, both individually and with our colleagues Carole Morrell and Mandy Morrell at University College Northampton; our findings will be reported in later articles.

References 1. 2. 3. 4. 5. 6. 7.

D. Kahneman, P. Slovic, and A. Tversky, eds., Judgement under Uncertainty: Heuristics and Biases, Cambridge Univ. Press, Cambridge, UK, 1982. E. Eich et al., “Memory for Pain: Relation Between Past and Present Pain Intensity,” Pain, vol. 23, 1985, pp. 375–389. E. Ohnhaus and R. Adlev, “Methodological Problems in the Measurement of Pain: A Comparison between the Verbal Rating Scale and the Visual Analogue Scale,” Pain, vol. 1, 1975, pp. 379–384. D. Price et al., “The Validation of Visual Analogue Scales as Ratio Scale Measures for Chronic and Experimental Pain,” Pain, vol. 17, 1983, pp. 45–56. S. Rachman and K. Eyrl, “Predicting and Remembering Recurrent Pain,” Behaviour Research and Therapy, vol. 27, no. 6, 1989, pp. 621–635. G. Gigerenzer, “Why the Distinction between Single Event Probabilities and Frequencies Is Important for Psychology (and Vice Versa),” Subjective Probability, G. Wright and P. Ayton, eds., John Wiley & Sons, Chichester, UK, 1994, pp. 129–161 G. Wright and P. Ayton, eds., Subjective Probability, John Wiley & Sons, Chichester, UK, 1994.

in this area based on an evaluator’s observation, which is standard practice in system development. During software use, the evaluator would estimate the user’s feelings for each screen or module, then note them on a standard Likert-style scale, with the aim of identifying peak–end values. This would not entail much extra effort, and our usual procedures could easily accommodate the change.

In pleasure and pain research, for example, the researcher must choose whether the scale should begin at zero (“no pleasure at all”) or with a minus value (“extreme pain”). As our own findings suggest, the wording and values used for Likert-style scales can affect the results. This is something we must consider seriously in any future work on the peak–end effect.

Limitations of Likert-style scales Although Likert-style scales have been in widespread use for decades, they have wellrecognized problems that have important methodological implications for anyone investigating the peak–end effect or conducting general usability research. One major problem is the potential for experimenter bias. Researchers must label each end of the scale, and these labels have significant implications for subjects’ likely answers.

Implicit attitudes Although we asked experiment participants to rate their feelings on a scale, repeated psychological research has found that people can hold implicit attitudes that differ significantly from those they’re consciously aware of. This has considerable implications for usability evaluation research (as well as for market research, where the problem is more widely known). The psychology literature describes various established methods May/June 2002

IEEE SOFTWARE

67

For safetycritical applications, we strongly advise the use of best-practice methods for acceptance testing.

68

IEEE SOFTWARE

for indirectly measuring implicit attitudes. Applying these methods to usability evaluation is an area for future research. Research implications Although preliminary, our results are significant and have implications for usability design and evaluation, as well as for areas such as prioritizing debugging efforts. We view all of the following implications as drivers for further research rather than as truths on tablets of stone. Ethical implications The ethical implications of our results could be considerable. An obvious concern is safety-critical software, where the peak–end effect might result in dangerously flawed software passing an acceptance test. Given the well-documented cases in which software flaws have led to deaths, such as in the Therac-25 system,9 the risk is nontrivial. Adherence to standard development and testing methodologies should prevent this problem. Testing methodologies are designed to produce a systematic testing process unskewed by subjective effects of the sort we describe. However, such methodologies are not uniformly applied; developers of the Therac-25 software, for example, did not follow a standard methodology, and the software apparently underwent no standard acceptance testing. In non-safety-critical areas, the risk of peak–end effect misuse is probably higher because of commercial pressure. The ethical issues here are also more complex than those involved in safety-critical systems. A developer might claim, for example, that it’s unethical to leave software users with anything but the most pleasant memory possible. Although this argument has some initial plausibility, we consider it flawed. Software developers need not choose between producing good software and abusing knowledge of the peak–end effect, nor is the peak–end effect an excuse for deliberately producing bad software. An unscrupulous developer might, for example, deliberately supply flawed software using the peak-end effect rather than fix an expensive and obscure bug. Our work suggests two overlapping ways to subjectively evaluate software quality: contemporaneous and retrospective. In our opinion, software developers have an ethical

May/June 2002

responsibility to produce software that is user friendly on both counts. Most design guidelines implicitly concentrate on contemporaneous evaluation and provide developers more than adequate guidance. Regarding retrospective evaluation, developers also have a responsibility to produce user-friendly software, but the dividing line between user friendliness and consumer manipulation is not always clear. In this area, we need informed debate among relevant professional bodies, along with best practice guidelines. To mitigate the potential ethical abuse of the peak–end effect, we have two recommendations. For safety-critical applications, we strongly advise the use of best practice methods for acceptance testing and considerable caution in using testing criterion that involves subjective opinions about the software as a whole. For non-safetycritical applications, we recommend adherence to standard best practice in software design and evaluation. Other implications Our results have implications for at least three other areas as well. ■

■

■

Debugging and usability design. Our results have implications for debugging strategy and for usability design that can be tested fairly easily. The peak–end effect implies that a single experience of a particularly unpleasant bug will leave a stronger impression on a user than several experiences of a bug that is not quite so unpleasant. Similarly, it implies that a single experience of a particularly pleasant design feature will leave a stronger impression than several experiences of a feature that is less pleasant. Measurement. The same principle likely applies to other aspects of customer contact, including measures of customer care and satisfaction. We are currently investigating customer satisfaction measures in an industrial case study. Methodological implications. As we discussed earlier, our investigation highlighted broader methodological issues— including implicit attitudes and the problems with Likert-style scales—that are particularly relevant to researchers in this area. They are also highly relevant to usability and market research in general.

More work is needed in all of the above areas to further understand the implications of such effects within the software engineering domain.

3.

4.

5.

A

s the “Judgment and Decision-Making Literature” sidebar describes, that field’s research contains many robust findings with implications for usability evaluation; the peak–end effect is just one of these. The psychology literature has given much attention to such issues, but the direction and nature of the effects are not yet well understood. Nonetheless, usability researchers are well advised to consider these issues and their implications for standard usability practice. Regarding our own future work, one priority is investigating data collection on an event-driven rather than a time-driven basis. Another priority involves clarifying the methodological issues involved in investigating the peak–end effect in the software domain. We also want to investigate the extent to which expressed attitudes, as measured in the experiments described here, correlate with actual behavior. For example, does a strong peak–end effect during a free product trial mean that users are more likely to buy the product? If the peak–end effect emerges as a strong influence on user behavior, then ethical and professional issues must be addressed. We hope this article will prove a useful starting point. In any case, users everywhere can take comfort in the fact that, although pain research might tell us much about software design, it seems we can learn even more from research into pleasure.

6. 7.

8.

9.

R. Molich and J. Nielsen, “Improving Human-Computer Dialogue: What Designers Know about Traditional Interface Design,” Comm. ACM, vol. 33, no. 3, Mar. 1990, pp. 338–342. N. Scott, An Investigation into Neophyte Software Users and Their Real-Time and Retrospective Evaluations, masters’ thesis, School of Information Systems, Univ. College Northampton, UK, Jan. 1999; for a copy, contact Sheila Guilford ([email protected]). D. Redelmeier, P. Rozin, and D. Kahneman, “Understanding Patients’ Decisions,” J. Am. Medical Assoc., vol. 270, no. 1, 1993, pp. 72–76. A. Baddeley, Human Memory: Theory and Practice, Lawrence Erlbaum Assoc., Hillsdale, N.J., 1990. A. Badii and A. Murphy, “PopEval_MB: A New Usability Evaluation Toolkit to Serve a System for IS/IT Cultural Match Management,” Proc. 6th European Conf. on Information Systems Evaluation, D. Remenyi, ed., Brunel Univ., Uxbridge, UK, pp. 352–368. B. Fredrickson and D. Kahneman, “Duration Neglect in Retrospective Evaluations of Affective Episodes,” J. Personality and Social Psychology, vol. 65, no. 1, 1993, pp. 45–55. N. Leveson and C.S. Turner, “An Investigation of the Therac-25 Accidents,” Computer, vol. 25, no. 7, July 1992, pp. 18–41.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

About the Authors Sheila Guilford is a senior lecturer in information systems at University College

Northampton. Her research interests are in human factors and human–computer interaction. She has an MS in man–computer systems from De Montfort University. Contact her at Univ. College Northampton, Park Campus, Northampton NN2 7AL, England; [email protected].

Gordon Rugg is a senior lecturer in computer science at Keele University, and was previ-

ously a reader in technology acceptance in the School of Information Systems, University College Northampton. He previously worked as a senior lecturer in the School of Computing Science, Middlesex University and is editor of Expert Systems: the International Journal of Knowledge Engineering and Neural Networks. He has a PhD in psychology from the University of Reading. Contact him at the Dept. of Computer Science, Keele University, Staffordshire ST5 5BG, England; [email protected].

References 1.

2.

D. Redelmeier and D. Kahneman, “Patients’ Memories of Painful Medical Treatments: Real-Time and Retrospective Evaluations of Two Minimally Invasive Procedures,” Pain, vol. 66, 1996, pp. 3–8. P. Polson et al., “Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces,” Int’l J. Man-Machine Studies, vol. 36, no. 5, 1992, pp. 741–773.

Niall Scott is a systems executive at Broadsystem in Manchester. His research interests in-

clude the application of psychology to real-world problems. He has a bachelor’s degree in psychology from the University of Newcastle Upon Tyne and a master’s degree in computing from University College Northampton. Contact him c/o Sheila Guilford at Univ. College Northampton, Park Campus, Northampton NN2 7AL, England; [email protected].

May/June 2002

IEEE SOFTWARE

69

feature traceability

Tracing All Around in Reengineering Gerald Ebner, Significant Software Hermann Kaindl, Siemens AG Österreich, PSE

egacy software must often be reengineered, a process that involves both reverse1 and forward engineering. Typically, no requirements, design, or design rationale documentation for legacy software exist. This lack of documentation means an unavailability of traces as well, making reengineering difficult and expensive. One way to improve reengineering efforts is to establish traceability—the ability to follow requirements information through the software life cycle—

L

Traceability in forward engineering has well-known long-term benefits. Tracing “all around” in reengineering, from code through specifications to code, can also yield immediate benefits, such as simplified requirements change and early error diagnosis. 70

IEEE SOFTWARE

during the reengineering process. We propose tracing “all around”—from existing code to the legacy system’s design to its requirements during reverse engineering, and from the modified requirements to the new design and the new code during development of the successor system. In effect, this means integrating traceability during reverse engineering with (more standard) traceability during forward engineering. Our approach uses the Requirements Engineering Through Hypertext (RETH) method and tool to install traces during both reverse and forward engineering.2,3 Our experience in a real-world reengineering project suggests that all-around tracing can yield immediate benefits that appear to be specific to reengineering. In this article, we describe all-around tracing and present a case study to demonstrate how our approach can be useful in reengineering efforts. Traceability in legacy software reengineering Installing traces takes time and can be costly (unless installation can be automated), and real-world projects are severely

May/June 2002

time- and resource-constrained. So, for a practitioner, an important question is whether and how traceability can immediately benefit a reengineering effort. To date, no studies have shown a positive trade-off between the cost and short-term benefits of traceability. We have found empirical evidence of short-term benefits and thus argue in favor of incorporating traceability throughout both legacy software reverse engineering and forward engineering. A high-level view To illustrate our high-level view of traceability in reengineering, we use a Unified Modeling Language (UML, see www.omg. org/uml) class diagram enhanced with arrows, as shown in Figure 1. In all-around tracing for reengineering, we trace from the old system implementation to its design and requirements during reverse engineering, and from the modified requirements to the new design and system implementation during forward engineering. Once we have explicitly established traceability from the old to the new implementation via design and requirements specifica0740-7459/02/$17.00 © 2002 IEEE

Explicitly installed traces during reverse and forward engineering Indirect traceability

tions, these implementations are indirectly connected. The arrow with the broken line in Figure 1 illustrates this relationship. From a theoretical perspective, the relationship is included in the transitive closure of explicitly represented traceability relations. In practice, this means that we do not need to represent an explicit relation between the implementations because we can derive it. In our view, we need only one requirements definition if the old and the new software have more or less the same requirements. Although reengineering projects in practice typically deal with at least minor changes or additions to the requirements, a single specification defining both old and new requirements can be sufficient. Metamodel for all-around tracing Several traceability metamodels exist in the literature; however, they are not sufficiently specific for immediate application in practice. (See the sidebar, “Related Work on Traceability.”) The detailed UML class diagram in Figure 2 illustrates our metamodel for representing requirements, design, and implementation, as well as the connections between them. This is the model of how the models look during reverse and forward engineering in our approach. We use RETH for requirements engineering and traceability. RETH is both a method and a tool that supports the method. The top section of Figure 2 (the yellow-green boxes, or metaclasses) is the RETH metamodel for requirements engineering.2,3 This part is aggregated in Figure 2 by requirements definition. The other classes extend this metamodel to include design (green boxes) and implementation artifacts (dark green boxes). Our extended metamodel also includes a simple representation of design rationale (design decision, which is part of design). Trace associations represent traces, which we install between requirements definition and design and between design and implementation. For the latter traces in particular, trace granularity is of practical importance. According to our experience, traces should not be too coarse—for example, to a source code module as a whole. But if a trace is too fine-grained—for example, to each statement in the source code—installing it is too expensive. For balance, we put traceability

Requirements definition

RETH hypertext link

Old system design

New system design

Traceability tag

Old system implementation

New system implementation

Figure 1. Tracing all around. With all-around tracing, we can trace from the old system implementation to its design and requirements, and then from the modified requirements to the new design and implementation.

tags at the procedure level. This requires reasonable effort and is still useful. We include a correspondence association between domain object and design object to support additional traces. In our approach, a domain object is an abstraction of an entity in the problem domain. While we can clearly distinguish between domain objects and (software) design objects,4 a correspondence between them can certainly exist. Our representation of design rationale is a practical compromise. Each design decision describes only the design chosen and the alternatives considered. It is unlikely that we could have maintained more elaborate structures within the project time constraints. It was important, however, that the design decisions be traceable. During reengineering, we develop two models based on this metamodel: one during reverse and one during forward engineering. More precisely, a common requirements definition links the models of the old and the new designs and implementations for allaround tracing (as Figure 1 illustrates). This approach is sufficient as long as the requirements are completely stable. We should anticipate, however, that requirements will change between the old and new systems. Figure 3 shows that the requireMay/June 2002

IEEE SOFTWARE

71

Related Work on Traceability Several articles in a special section (edited by M. Jarke) of the Communications of the ACM present high-level proposals for traceability.1,2 Still, we found little in the literature about the trade-off between the cost and utility of traceability, and in particular about its short-term benefits. In current practice, however, it is very important to show developers that rewards can be immediately gained from installing traces. Finally, we found no mention of the specific utility of tracing during reengineering or even of all-around tracing.

ments,” instead of relating concrete design and requirements artifacts. Our metamodel defines in greater and more concrete detail specialized metaclasses and their relationships. In this sense, we think that our metamodel is easier to apply. In their case study, Ramesh and colleagues report that installing traces according to their metamodel is very expensive,5 but they do not report short-term benefits or the specific utility of traceability in reengineering.

Design decision support Traceability tools

A great deal of literature on design rationale exists,5,6 most of it proposing quite elaborate argumentation structures. BalaTraceability among requirements, primarily from higher- to lower-level requirements, has long been considered and is well subramaniam Ramesh and Vasant Dhar, for example, present a conceptual model for recording discussions and decisions supported by tools. The Automated Requirements Traceability about requirements in a network of issues, positions, and arSystem (ARTS), for example, is a software database manageguments connected by special links.6 We could easily reprement system for requirements information management.3 Several tools for requirements management and traceability based sent such structures using hypertext in the RETH tool. They on database systems are commercially available—for example, would have been too expensive to maintain under the Trade Requirements Traceability and Management (RTM, www. Bridge project conditions, however. As a practical comprochipware.com), Dynamic Object-Oriented Requirements mise, we included a simple representation for design rationale System (DOORS, www.telelogic.com), and RequisitePro and integrated it with the traces. (www.rational.com). Of course, such tools could have managed traces in the Trade Bridge system as well as the RETH tool did. One of the References 1. G. De Michelis et al., “A Three-Faceted View of Information Systems,” main reasons developers chose RETH instead of one of these Comm. ACM, vol. 41, no. 12, Dec. 1998, pp. 64–70. COTS tools was its automatic link-generation facility. Reflecting 2. R. Dömges and K. Pohl, “Adapting Traceability Environments to Projectmore recent research, Traceability of Object-Oriented RequireSpecific Needs,” Comm. ACM, vol. 41, no. 12, Dec. 1998, pp. 54–62. ments (TOOR) treats requirements and their relations as objects 3. R.F. Flynn and M. Dorfman, “The Automated Requirements Traceability System (ARTS): An Experience of Eight Years,” in R.H. Thayer and M. Dorfsimilarly to our approach.4

Metamodels for traceability Balasubramaniam Ramesh and colleagues present a requirements traceability metamodel that also covers design and implementation.5 It is more comprehensive than ours because it includes, for example, stakeholders. It is rather high-level, however. For example, it simply states “design satisfies require-

man, eds., System and Software Requirements Eng., IEEE CS Press, Los Alamitos, Calif., 1990, pp. 423–438. 4. F.A.C. Pinheiro and J.A. Goguen, “An Object-Oriented Tool for Tracing Requirements,” IEEE Software, vol. 13, no. 2, Mar. 1996, pp. 52–64. 5. B. Ramesh et al., “Requirements Traceability: Theory and Practice,” Annals Software Eng., vol. 3, Sept. 1997, pp. 397–415. 6. B. Ramesh and V. Dhar, “Supporting Systems Development by Capturing Deliberations During Requirements Engineering,” IEEE Trans. Software Eng., vol. 18, no. 6, June 1992, pp. 498–510.

ments definition can include common re-

quirements as well as requirements specific to one system. Note that not all of these must exist in practice. Often the majority are common requirements, and only a few are new system requirements. When one or more new requirements replace one or more old requirements, we associate them with respect to traceability (see Figure 3). Implementing traceability In our case study, the requirements engineer and chief designer (the first author of this article) used the RETH tool not only for requirements capture, but also for software design (both old and new) and traceability implementation. Although the RETH method as previously published does not cover software 72

IEEE SOFTWARE

May/June 2002

design, the approach and its supporting tool are flexible and extensible, simplifying the extensions to the metamodel in Figure 2.2 Hyperlinks in the RETH tool let us install traces by linking various artifacts in the requirements specification to artifacts in the two software designs (illustrated in Figure 1). The RETH tool’s semiautomatic support for link generation makes installing these hyperlinks inexpensive.5 The mechanism for generating glossary links can also semiautomatically generate traceability links based on textual references. An immediate advantage of such links is that users can easily navigate them to follow traceability paths. This navigation is possible in both directions because RETH hyperlinks are bidirectional. For technical reasons, the source code of

Figure 2. Metamodel for representing requirements, design, and implementation and the connections between them. We install traces between requirements and design artifacts and design and implementation artifacts. A correspondence association supports additional traces between domain objects and design objects.

<<stereotype>> Requirements definition

+is similar to <<stereotype>> Goal

+is achieved by

<<stereotype>> Domain object

<<stereotype>> Requirement Statement Further explanation Source Priority Reason(s)

Statement

Trace

Description

+conflicts with

Correspondence

+justifies +achieves +makes possible <<stereotype>> Functional requirement

<<stereotype>> Scenario

<<stereotype>> Quality requirement

+fulfills +constrains

+constrains

+justifies <<stereotype>> Design

+considers

+considers <<stereotype>> Design goal Statement

<<stereotype>> Design scenario

<<stereotype>> Design decision

<<stereotype>> Design object

+affects Design chosen Statement Description Alternatives considered Alternatives considered

Trace

Trace <<stereotype>> Source code file

Trace

Trace

Trace

<<stereotype>> Implementation

Trace

<<stereotype>> Database script

Figure 3. Requirements change. In our metamodel, the requirements definition allows for common requirements as well as requirements specific to the old or new system.

<<stereotype>> Requirements definition

<<stereotype>> Old and new system requirement

<<stereotype>> Old system requirement only

+replaces

the old and new software had to remain outside this tool, which made it infeasible to install RETH hyperlinks to or from source code. To compensate, we use traceability

<<stereotype>> New system requirement only

tags between design artifacts and source code (see Figure 1). A traceability tag in our case study is simply a hexadecimal number surrounded by May/June 2002

IEEE SOFTWARE

73

Mid-office system

1: Query Front-office system

Trade Bridge

2: Front-office data

Control flow

Risk-management system

Ticket printer

Back-office system

Data flow

Figure 4. Trade Bridge collaborations with other software. Trade Bridge receives data from the front-office system. It checks this data for consistency, then sends it to the midoffice, risk-management, and back-office systems and the ticket printer.

the special character ° (for example, °34F5°). Inserting the same tag at all the places in the various documents to be linked together can implement binary as well as higher-order relations, but a single tag by itself does not indicate traceability direction. From a practical viewpoint, this does not pose real problems because the metamodel clearly shows the direction. We implemented a simple tool to generate and then find unique tag names. We can easily find tags in virtually any kind of document as long as the corresponding MS Windows tool provides full-text search. Further support for renaming, deleting, and checking tags might be desirable, but even the simple tool support available in our project was useful. A real-world case study Our case study involved the reengineering of Trade Bridge, a real-world legacy software system that continuously checks the consistency of received stock data and distributes that data to other software systems. The old Trade Bridge system was unreliable, difficult to maintain, and based on old technology. Because users wanted additional functionality, a new manager decided to reengineer the system. The project, from kick-off to full operation, lasted approximately one year. 74

IEEE SOFTWARE

May/June 2002

During the project, developers installed and used traceability in real time. Here, we describe the task, the project’s basic reengineering approach, and some experiences, focusing exclusively on traceability and its benefits for developers of the new software. Figure 4 illustrates on an abstract level how Trade Bridge interfaces with other software systems. The front-office system delivers trade, position, and other data to Trade Bridge, which stores the data in a repository (implemented as a relational database). After checking the data for consistency, Trade Bridge sends the checked data to the midoffice system, the risk-management system, the ticket printer, and the back-office system. Basic reengineering approach Immediately after the reengineering project began, the requirements engineer acquired information about the old software requirements from its developers. Fortunately, this was possible, and it saved some requirements and design recovery effort. Later, when the developers of the old software were no longer available, the reengineering team had to recover everything (including requirements). Our approach does not prescribe a process that requires either the original system developers or documentation. Rather, it defines a metamodel for representing useful information. The requirements engineer/chief designer first represented the old software requirements in the RETH tool. In his design role, he tried to figure out how these requirements related to the implementation. He represented the resulting design information and the design rationale—which he partly acquired from the developers of the old system and partly hypothesized—in the RETH tool (compliant with the metamodel in Figure 2). During both reverse and forward engineering, he immediately installed traces as hypertext links and traceability tags. There were immediate benefits to having these traces available. Handling requirements change In the Trade Bridge project, traces facilitated the handling of requirements change with respect to related design decisions. One requirement, for example, concerned the accessibility of configuration parameters, which influences their storage method in Trade

Table 1 Conflicting requirements of the old and new software systems Requirement

Old system

New system

Functional

Easy access to configuration parameters: “Trade Bridge shall allow for easy change of its configuration parameters.”

Restricted access to configuration parameters: “Trade Bridge configuration parameters shall be accessible for change to system administrators only.” Hierarchical structure of configuration parameters: “To give system administrators an overview of the configuration parameters, the parameters shall be represented and displayed hierarchically.”

Quality

Table 2 Arguments for and against two designs for the new software Design decision: Storing configuration parameters (new) Design chosen: Store configuration parameters in the MS Windows registry Pros Cons

Alternative considered: Store configuration parameters in a relational database Pros Cons

For MS Windows NT (as used Providing context-sensitive online for Trade Bridge), only users help is difficult. with administration rights can change the registry. Programs can easily access the registry. System administrators are already familiar with the registry. The registry is already available.

Context-sensitive online help could be easily provided.

Bridge. The chief designer reverse-engineered the design decision, “storing configuration parameters,” from the old Trade Bridge software code, with the design chosen, “configuration parameters are generally stored in ASCII files, or, more precisely, in MS Windows INI format.” Fortunately, the old system developers were still available for an interview, which revealed the functional requirement (see Table 1) that justifies this design. We represented this requirement, the related design decision, and the link between them in the RETH tool compliant with our metamodel. No other requirements or design aspects appeared to be related to this functional requirement. Later, during forward engineering, users of the new software required different functionality. The requirements engineer formulated the new functional requirement, which Table 1 also shows. By this time, the developers of the old Trade Bridge software had left the company and the project. Nonethe-

Implementation of the database and access to it would require extra effort.

Implementation of a hierarchical representation would be particularly expensive. The database would need extra administration during use.

less, the RETH tool traceability link meant that the old design decision was just a hyperlink away from the old requirement. In this example, the old and the new requirements conflict. The new requirement therefore replaces the original, which the corresponding association in Figure 3 represents, thereby facilitating requirements change traceability. A new quality requirement related to the configuration parameters (see Table 1) also influenced the new design. The requirements change made it impossible to keep the old design, which presumably only the replaced requirement justified. No other design aspect was related to the replaced requirement. Therefore, the new design decision (see Table 2) only considered the new requirements (see Table 1), and it was linked to them to facilitate traceability. As illustrated in Table 2, the design chosen was clearly preferable to the alternative design. When requirements change, developers must determine the consequences of this change on the design. In reengineering, May/June 2002

IEEE SOFTWARE

75

function MakeInstrumentQuery(): string; // determine which instrument data to request from the // front-office system and build the “where” part of // the query string °41° begin ... SqlExecute(‘select distinct ID into strID from Instrument_Get where tries < 10 order by tries’); while SqlFetch() and (nLines < 5) do begin strQuery = strQuery + strID + ‘, ‘; nLines = nLines + 1; end; if (nLines > 0) then ... end;

Figure 5. Code from the old procedure. The old software requested only five data records at a time from the collaborating front-office system.

knowing which design decisions are related to replaced requirements is important. Developers should reconsider these decisions when designing the new system. Knowing the rationale behind old decisions can usually help them make better new decisions. In this way, traces installed during reverse engineering can assist developers during the forward-engineering process. This is one example of the short-term benefits of traceability. In particular, this example shows how the following features of our traceability approach can be useful. ■ ■ ■

We establish traces during reverse engineering (Figure 1). Requirements justify design decisions (Figure 2). New requirements replace old ones (Figure 3).

Diagnosing reimplementation errors In the Trade Bridge project, traces also facilitated diagnosis of reimplementation errors. After implementation of the new Trade Bridge software, tests revealed a performance problem. The front-office system could not deliver the amount of data the new software requested—a problem that hadn’t occurred with the old Trade Bridge software. During redesign and reimplementation, developers had used different names for artifacts and had distributed functionality dif76

IEEE SOFTWARE

May/June 2002

ferently. Finding the corresponding parts in the old and the new implementations could therefore be tedious and time consuming. The installed traces, however, made it easy to find the corresponding parts in the old and new implementations immediately using our tool support for finding traceability tags, and thus helped determine the problem. The old software avoided the data-delivery problem by requesting only five data records at a time from the collaborating front-office system. This fact was well hidden in the old procedure’s code, as Figure 5 shows. The critical condition is and (nLines < 5). This is clearly below tag granularity, where °41° is the procedure traceability tag. (In several cases not reported in this article, the granularity of installed traceability tags was sufficient.) Having the traces in place let us correctly diagnose the problem within one hour; with another method, such as searching artifact names, it might have taken days. Thus, traces can also help solve problems resulting from incomplete reimplementation. In particular, this example shows how using our approach for all-around tracing in reengineering can be useful: establishing traces during reverse and forward engineering provides (derived) indirect traceability between the old and new implementations (see Figure 1). Lessons learned Our experience with the Trade Bridge reengineering effort made clear several benefits of implementing traceability in both reverse and forward engineering. ■

■

■

Traces to the reverse-engineered requirements and design information can facilitate new software development within the same reengineering effort, if they are installed immediately. Although elaborate design rationale might be too costly to include, simple information on design decisions is feasible to maintain and trace in practice, even under heavy time constraints, making traceability even more useful. A trade-off in trace granularity exists. It relates to the more general trade-off between the cost of installing and maintaining traces and the benefit of having them available. Common wisdom suggests that traceability should pay off in

the long run, which would include maintenance. Unfortunately, whether a software development or reengineering project provides traces to facilitate later maintenance does not usually determine its success or failure in practice. Budget and time constraints are major issues in any project. Project budgets often cover only the cost of delivering a running system; costs arising later due to missing traces are outside their scope. From a higher perspective, this is both shortsighted and regrettable. In addition, many software systems are developed but never deployed, so developers focus on delivering running systems on time. Preparing for an uncertain future is less important: other people will be responsible for any maintenance costs. We therefore argue for a distinction between immediate and long-term benefits. Although we cannot provide quantitative data to show that traceability paid off within the Trade Bridge reengineering effort, it did provide benefits over the course of the project. In fact, other cases in the Trade Bridge project also support traceability’s usefulness in the short term. These cases were particularly related to ■ ■ ■ ■

I

Additional requirements Software design changes Improved code completeness Iterative and incremental reverse engineering

n all cases, developers were immediately rewarded for their trace installation effort, which further motivated them to do this “extra” work. This situation is different from most development efforts with separate maintenance. Whenever a single team is responsible for all aspects of a reengineering project, the team can quickly harvest the fruit of “seeding” traces. Such short-term benefits can motivate them

About the Authors Gerald Ebner is managing partner of Significant Software, an Austrian consulting and

software development company that focuses on e-business and MIS projects, as well as scientific software. His research interests include requirements engineering with a focus on traceability and the documentation of design rationale. He received the Dipl.-Ing. in electrical engineering from the Technical University of Vienna. He is a member of the ACM. Contact him at Significant Software, Zeltgasse 14, A–1080 Vienna, Austria; [email protected].

Hermann Kaindl is a senior consultant at Siemens AG Österreich, PSE. His research in-

terests include software engineering, with a focus on requirements engineering; human-computer interaction as it relates to scenario-based design and hypertext; and artificial intelligence, including heuristic search and knowledge-based systems. He received the Dipl.-Ing. in computer science and his doctoral degree in technical science, both from the Technical University of Vienna, where he has lectured since 1984 and served as an adjunct professor since 1989. He is a senior member of the IEEE and a member of the ACM, and is on the executive board of the Austrian Society for Artificial Intelligence. Contact him at Siemens AG Österreich, PSE, Geusaugasse 17, A–1030 Vienna, Austria; [email protected].

to provide traces that might also result in further long-term benefits later, possibly for other people than the developers.

Acknowledgments We thank the project members involved in the Trade Bridge reengineering effort for their cooperation. Mario Hailing and Vahan Harput provided useful comments on an earlier draft of the article. Mark Sinclair made grammatical corrections. Finally, we acknowledge the very useful comments of the anonymous reviewers, who identified several weaknesses in an earlier version of this article.

References 1. E.J. Chikofsky and J.H. Cross, “Reverse Engineering and Design Recovery: A Taxonomy,” IEEE Software, vol. 7, no. 1, Jan./Feb. 1990, pp. 13–17. 2. H. Kaindl, “A Practical Approach to Combining Requirements Definition and Object-Oriented Analysis,” Annals Software Eng., vol. 3, Sept. 1997, pp. 319–343. 3. H. Kaindl, “A Design Process Based on a Model Combining Scenarios with Goals and Functions,” IEEE Trans. Systems, Man, and Cybernetics (SMC), Part A, vol. 30, no. 5, Sept. 2000, pp. 537–551. 4. H. Kaindl, “Difficulties in the Transition from OO Analysis to Design,” IEEE Software, vol. 16, no. 5, Sept./Oct. 1999, pp. 94–102. 5. H. Kaindl, S. Kramer, and P.S.N. Diallo, “Semiautomatic Generation of Glossary Links: A Practical Solution,” Proc. 10th ACM Conf. Hypertext and Hypermedia (Hypertext 99), ACM Press, New York, 1999, pp. 3–12.

For more information on this or any other computing topic, please visit our digital library at http://computer.org/publications/dlib.

May/June 2002

IEEE SOFTWARE

77

feature

software components

How Software Patents Can Support COTS Component Business Michael S. Guntersdorfer, Duke University David G. Kay, University of California, Irvine

Commercial-off-the-shelf software components are an important technology for software oftware components are on the rise. They form the basis of many well-designed modern software systems, provide a suitable level of granularity for software reuse, and are even a sales unit themselves, traded off-the-shelf like complete software systems.1 However, compared to its potential, component vending is still unprofitable. The software community appears to blame this on missing, insufficient, or incompatible interaction standards, leading to the oft-cited problem of

S Commercial offthe-shelf software components are an important technology for software development. This article discusses how software patents can strengthen COTS software’s weak business model and lead to improvements in existing technology. 78

IEEE SOFTWARE

“architectural mismatch.”2 However, the absence of a clear business model also contributes to the lack of profitability (as Butler Lampson mentioned in his presentation at the 21st International Conference on Software Engineering). This article discusses how software patents can support the software component business model and suppress multiple standards. These standards are due to marketing strategies in which each vendor promotes standardization as long as the standard is its own. We advocate neither the COTS approach nor software patents per se. Others have advanced or critiqued both positions (see the “Further Reading” sidebar). Rather, we focus on the interaction between patents and components. We argue that software patents, now a permanent facet of the business landscape though not yet universally employed, can support the commercial

May/June 2002

adoption of component-based software. This salutary effect of software patents cannot be demonstrated within one organization or for one product. It is a systemic change that will take place only as the entire patent system develops further and is applied regularly to software components. Why patent software? The three major mechanisms that US law offers to protect intellectual property are trade secrets, copyrights, and patents. (In this article, we consider primarily US law. With increasing globalization, however, the laws of other countries and the international agreements that attempt to reconcile them are becoming increasingly relevant.) A trade secret is any information a business uses that is not generally known. The computer industry has traditionally relied on trade secret protection of the source 0740-7459/02/$17.00 © 2002 IEEE

Further Reading

code for mass-market software by selling that software only in object code form (possibly encrypted to hamper reverse engineering by decompilation or disassembly). Trade secret law prohibits people with access to the secret from divulging that secret without authorization. However, it does not protect against the independent discovery or creation of the same information, or even against reverse engineering the information from publicly available sources. Trade secrets are a method of intellectual property protection that is hard to suppress. Delivering only object code hinders innovation by hiding innovative concepts. This is the antithesis of reuse. Copyrights protect original works of authorship against copying—works such as prose, drama, music, art, and software, whether published or unpublished. A copyright does not protect against independent creation of the same information. It does not protect the underlying ideas in a work; it only protects the particular way those ideas are expressed. This protection goes beyond the literal; copyright also protects derivative works, such as those translated to other programming languages or platforms. Similarity clearly indicates copying in creative works such as fiction and music, where most of the work is determined only by the author’s creativity and is subject to few practical constraints. But for works with a functional aspect, such as software, that functionality is part of the work’s underlying idea and is not subject to copyright protection. Software with similar functionality to a copyrighted work, but created independently, does not infringe. Reliance on copyright protection alone does nothing to curtail independent development of functional equivalents. Moreover, the copyright term— nearly a century—is completely foreign to the time scale of software development. Trade secrets and copyrights protect primarily the creator’s lead time, forcing subsequent implementers to do their own work rather than piggyback on the creator’s effort. Stemming the proliferation of similar components requires stronger, deeper protection. Patents protect inventions, not just specific embodiments or implementations. A US patent gives an inventor up to 20 years from the filing date (subject to certain adjustments) to exclude others from making, us-

Information on commercial-off-the-shelf components is available at wwwsel.iit.nrc.ca/projects/cots/icse2000wkshp. For the pros and cons of software patents, we recommend these publications: L. Graham, “Debunking Software Patent Myths,” IEEE Software, vol. 17, no. 4, July/Aug. 2000, pp. 122–123. ■ P. Heckel, “Debunking the Software Patent Myths,” Comm. ACM, vol. 35, no. 6, June 1992, pp. 121–140. ■ League for Programming Freedom, “Against Software Patents,” Comm. ACM, vol. 35, no. 1, Jan. 1992, p. 17. ■ T. O’Reilly, “The Internet Patent Land Grab,” Comm. ACM, vol. 43, no. 6, June 2000, pp. 29–31. ■

ing, offering for sale, or selling the invention. As a practical matter, this gives the patent holder control over how others employ the invention. Rather than simply barring all others, the patent holder will typically grant licenses on payment of a royalty. These licenses are contracts that can restrict modification, require adherence to standards, or otherwise further the inventor’s aims. To receive this protection, an invention must be novel and not merely an obvious extension of the “prior art” (inventions that are already patented, published, in use, or known). The invention must also involve the appropriate subject matter: “any new and useful process, machine, [article of] manufacture, or composition of matter, or any new and useful improvement thereof.”3 Patent law does not apply to laws of nature, scientific phenomena, or mathematical formulas,4,5 because awarding exclusivity to such concepts (which arguably have always existed) would grant control over the physical world that individuals should not reasonably exercise. Patents can protect truly novel software ideas from unfair exploitation, while releasing the knowledge itself to the community. They work against secrecy and support scientific progress, which, in the case of software, includes software reuse. Historically, patents have worked with new technologies in this way, and we would not expect software to be an exception. How software became patentable Early US Supreme Court decisions ruled that software consists essentially of mathematical algorithms and was thus not patentable.6,7 Between 1972 and 1981, the computer industry developed rapidly. Without the US May/June 2002

IEEE SOFTWARE

79

In 1981, the US Supreme Court ruled that the Patent Office could not deny a patent solely because it contained a computer program.

Patent Office acting during this period as a repository for the prior art of software—applications, coding techniques, algorithms, and data structures—ascertaining the novelty of software patent applications was difficult. Results in the quality of issued patents have been mixed, causing uncertainty in the software industry. (Because the 20-year mark since 1981 has passed and the repository of prior art is becoming more developed, we would expect these problems to diminish.) In 1981, the US Supreme Court ruled that the Patent Office could not deny a patent solely because it contained a computer program.8 After that, the US Patent Office developed guidelines and began issuing patents that were explicitly software-related. Another milestone was cleared in 1998, when the Court of Appeals for the Federal Circuit held that business methods were patentable on the same basis as other methods, and that mathematical algorithms can be patented unless they represent only abstract ideas.9 This overruled a district court case equating software with mathematical algorithms, essentially putting an end to both the business method and the mathematical algorithm exceptions.10 The 1990s also brought changes on international grounds: Effective 1 January 1995, the Uruguay Round of negotiations of the World Trade Organization revised the General Agreement on Tariffs and Trade. The WTO amended the Agreement on Trade-Related Aspects of Intellectual Property Rights (the TRIPS Agreement), which regulates standards, enforcement, and dispute settlement of intellectual property rights, including patents, to make patents available for any inventions, whether products or processes, in all fields of technology, provided that they are new, involve an inventive step, and are capable of industrial application.11

The amended agreement also required that patents…be available and patent rights enjoyable without discrimination as to the place of invention, the field of technology, and whether products are imported or locally produced.11

At the time of publication, the WTO had 144 member countries, all bound to its agreements. (For a list of members, see www.wto. org/english/thewto_e/whatis_e/tif_e/org6_e. 80

IEEE SOFTWARE

May/June 2002

htm.) The TRIPS Agreement is therefore likely to significantly affect intellectual property protection worldwide. Similarities in the wording of the GATT and US patent law are apparent. So, it is reasonable to believe that software patents eventually will be an accepted common practice internationally. Why patent components? Two main issues hinder componentbased software reuse. One is technical: Software component interaction is implemented by multiple standards. The other is economic: It is difficult to identify and quantify the exact economic benefit of developing reusable software components. Multiple standards The problem with multiple standards is that a component must support several (or all) of them or it will enjoy only a fraction of the market. Unfortunately, no recovery is in sight. The Object Management Group promotes the continuously changing and growing Corba specification.12 Sun Microsystems offers Java-based technologies, such as Enterprise Java Beans (EJB) and Java 2 Enterprise Edition (J2EE).13 And Microsoft Corporation has produced its own suite of technologies, from OLE (Object Linking and Embedding) via ActiveX14 to Component Object Model and DCOM (Distributed COM)15 to its latest offering, SOAP (Simple Object Access Protocol).16 Many of these technologies or “standards” use similar underlying methods. Java RMI (remote method invocation) and Microsoft’s COM use remote procedure calls (RPCs). COM also serves as the foundation for ActiveX and OLE.17 Java Beans uses the more decoupled and asynchronous approach of event-based interaction. Corba, traditionally trying to subsume all technologies in one, stays ambivalent and offers both RPC- and event-based interaction. Some of these methods could have been patentable along the lines of Sun Microsystems’ patent for “methods for routing events among publishers and subscribers on a computer network,” which claims a “method … for publishing and receiving events to and from a network” (US Patent No. 6,021,443). If these methods had been patented, the patent holders could have used their influence to reduce the proliferation of different

standards that use the same underlying methods. Merely imitating the patented technologies would have been infringement. If there were an original inventor of the concept of component-based interaction, that inventor might even have patented the general concept before particular interaction technologies were introduced. The patent holder could have licensed this broader, conceptual invention to developers of particular interaction technologies (who would still be able to obtain patents for their specific inventions). The patent holder could then have influenced which technologies the developers introduced on the market and in which field they introduced those technologies. Patenting successful methods might work in two distinct, positive ways. First, competitors might settle for licensing the patented technology, concentrating on its application rather than its reinvention in a minimally different form. This would lead to fewer competing standards (addressing the technical issue). At the same time, the patented technology would be used more frequently, leading to more user testing, more feedback, and, as a result, improved implementation of the technology. Second, competitors might decide to focus on developing truly innovative solutions patentable in their own right, rather than merely incremental tweaks. Both approaches could lead to better scientific progress than the current approach of imitating existing technology. Economic benefits Component development is not obviously profitable for two reasons. First, a component’s development costs increase or its target market decreases with every additional, newly established interaction technology. Fewer, better-differentiated technologies, which patents would encourage (as we noted earlier), could ameliorate this situation. Second, factors in the software development process make reuse unattractive. Project management often chooses cheap reimplementation of existing functionality over buying expensive licenses that might also regulate the use of the purchased software component. Coders gratefully comply, because they prefer designing from scratch to working their way through other programmers’

code. They also know that their importance to management as individual coders increases as the code’s idiosyncrasy increases. Dependence on individual coders decreases, however, if the system’s code is an assembly of standardized and reused components. An idiosyncratic software system’s inferior quality, which causes higher maintenance costs for the system later, is frequently overlooked or ignored in initial development. From the original developers’ view, these costs affect the maintenance and product support personnel rather than themselves. Software patents would prohibit an unlicensed third party from reengineering a protected software component’s functionality. Moreover, license terms could require licensees to use the purchased component without modifying its code. The idea to use software patents as a tool to facilitate software reuse is not entirely new.18 The argument essentially is that copyright-protected code (predominant in the 1980s) penalizes reuse but not reimplementation. Patents, however, prohibit unlicensed reimplementation while supporting licensed code reuse. If software patents support software reuse in general, it is only rational to believe that software patents will also support component-based reuse and, ultimately, a viable software component business.

Software developers often complain that honoring patents would limit their creative freedom.

Arguments against patents Software patents have been controversial in the software community. Software developers often complain that honoring patents would limit their creative freedom. Although patents certainly impose constraints on development, they also introduce a new freedom. Today, the constraints of time-tomarket software development strategies severely curtail the development process. The results are lower-quality products. Patent protection buys inventive programmers lead time that lets them produce higher-quality products using better but more time-consuming software engineering methods. Because software was not patentable early in its history, software patent opponents argue that technology has evolved differently and that software, therefore, does not fit the patent model. This argument will certainly lose weight as time passes. One hundred years from now, it won’t matter that few software inventions were patented May/June 2002

IEEE SOFTWARE

81

Software professionals should consider how to achieve the most effective results within the existing legal framework.

in the 1970s. Even though wheels and axles existed long before patents in any form, a modern automobile contains hundreds of patented inventions reflecting more recent innovations. A particular technology such as software need not “fit” the patent system, because patents are designed to fit the very concept of invention. Others object to the cost of patent infringement actions. Lawsuits are indisputably expensive, but few infringement questions proceed to the courtroom, much less to a full-blown trial. In most cases, a comparison of the patent and the allegedly infringing work will produce a clear result one way or the other. Rational business people acting on the advice of counsel will not pursue a hopeless case. Moreover, the justice system, by means of mandatory settlement conferences and injunctions, summary judgments, or other immediate relief, helps curtail the irrational. The legal system, whatever its costs and flaws, exists to resolve certain business disputes no matter what form they take. Patent infringement actions are not unique. Some suggest eliminating intellectual property protection for software entirely. We could rely on open-source principles, which would limit revenues to those gained from maintenance and support. This model has its attractions, but it has not been proved viable for all segments of the software market. Software evolution Software evolves quickly. By the time one version of a product is shipped, developers are already working on the next, less buggy, better-performing, more feature-laden version. The process of patent application and issuance is slow in comparison. It currently takes two to three years for a software patent to issue; by that time, the software invention might already be outdated. How could this problem be addressed? A patent already protects inventors in a limited way immediately after it is filed, although potential infringers cannot be prosecuted before the patent is granted. As soon as an inventor files a patent application, it keeps others from seeking a patent for the same invention. For improvements, software inventors simply file new patents, building on the old ones. A typical example is Microsoft’s

82

IEEE SOFTWARE

May/June 2002

“method of encoding arbitrary shapes” (US Patent No. 5,764,814), which they later improved by “separating the encoding of shape and texture” (US Patent No. 5,946,419). An inventor can secure legal protection at low cost early in the development cycle with a provisional patent application. Provisional applications, first authorized in June 1995, have few formal requirements for describing the invention, and inventors can file them quickly and inexpensively. Software engineers can also expect legislation to address their needs. The recently established Patent Term Guarantee Act of 1999, for example, extends patent terms in proportion to delays by the patent office— an attempt to make the office process patent applications quickly. Other proposals include lowering the burden of proof required to invalidate patents in court and replacing the US first-to-invent rule with the first-tofile rule more prevalent overseas (defining the inventor as whoever first files a patent application for the invention). These steps might seem small, but “they reflect an effort to adjust the patent laws to keep pace with the technology they protect.”19

I

n the final analysis, any radical change in the current applicability of patent law to software is unlikely. Software professionals should consider how to achieve the most effective results within the existing legal framework. (“Code reuse” might be valued in the software community, but in the legal community it is a core principle.) By working with the legal profession to supply the appropriate technical guidance, software professionals can shape the outcomes that will most benefit the entire software industry.

Acknowledgments We thank David Rosenblum of the Department of Information and Computer Science at the University of California, Irvine, for fruitful discussions regarding the issues covered in this article. We also thank Jeffrey Voas of Cigital for his input and support and for initiating the contacts that led to this publication.

About the Authors Michael S. Guntersdorfer is a Juris Doctor student at the Duke University School

of Law, where he serves as a staff writer on the Duke Law and Technology Review. He is always on the quest for synergetic effects between business, law, and software development and is particularly interested in software processes, object-oriented programming, and componentbased reuse as well as business models and intellectual property law. He holds a Diplom-Informatiker Universität from the Technical University of Munich and an MS in information and computer science from the University of California, Irvine. Contact him at [email protected]; www.duke.edu/~msg5.

References 1. V. Trass and J. van Hillegersberg, “The Software Component Market on the Internet: Current Status and Conditions for Growth,” Software Eng. Notes, vol. 25, no. 1, Jan. 2000, pp. 114–117. 2. D. Garlan, R. Allen, and J. Ockerbloom, “Architectural Mismatch or Why It’s Hard to Build Systems Out of Existing Parts,” Proc. 17th Int’l Conf. Software Eng., IEEE CS Press, Los Alamitos, Calif., 1995, pp. 179–185. 3. Inventions Patentable, US Code, vol. 35, sec. 101, 1994. 4. Mackay Radio & Telegraph Co. v. Radio Corp. of America, 306 US 86, 1939. 5. Funk Bros. Seed Co. v. Kalo Co., 333 US 127, 1948. 6. Gottschalk v. Benson, 409 US 63, 1972. 7. Parker v. Flook, 437 US 584, 1978. 8. Diamond v. Diehr, 450 US 175, 1981. 9. State Street Bank & Trust Co. v. Signature Financial Group, 149 F. 3d 1368, Fed. Cir., 1998. 10. M. Chu and P. Wied, “Avoiding Patent Potholes,” (presented at 20th Ann. Computer Law Inst.), Univ. of Southern California, Los Angeles, 1999. 11. General Agreement on Tariffs and Trade: Agreement on Trade Related Aspects of Intellectual Property Rights, Uruguay Round negotiations (1986–1994), World Trade Organization, Geneva, 1 Jan. 1995, Article 27.1, www.wto.org/english/docs_e/legal_e/27-trips.pdf. 12. T. Mowbray and R. Zahavi, The Essential CORBA: Systems Integration Using Distributed Objects, John Wiley & Sons, New York, 1995. 13. V. Matena and B. Stearns, Applying Enterprise Java Beans, Addison-Wesley, Reading, Mass., 2000. 14. D. Chappell, Understanding ActiveX and OLE, Microsoft Press, Redmond, Wash., 1996. 15. S. Willliams and C. Kindel, The Component Object

SET INDUSTRY STANDARDS

David G. Kay is a lecturer in the Department of Information and Computer Science at the University of California, Irvine. He teaches courses in computer law, programming languages, introductory computer science, and the social impact of computing. He holds a BA in linguistics and an MS in computer science from UCLA and a Juris Doctor from Loyola Law School. He is a member of the ACM, American Bar Association, and Computer Law Association. Contact him at the Department of Information and Computer Science, University of California, Irvine, CA 92697-3425; [email protected].

16. 17.

18.

19.

Model: A Technical Overview, white paper, Microsoft Corp., Redmond, Wash., 1994. D. Chappell, “Simple Object Access Protocol (SOAP),” white paper, Microsoft Corp., Redmond, Wash., 1999. K. Sullivan, J. Socha, and M. Marchukov, “Using Formal Methods to Reason about Architectural Standards,” Proc. 19th Int’l Conf. Software Eng., IEEE CS Press, Los Alamitos, Calif., 1997, pp. 503–513. M. Lemley and D. O’Brien, “Encouraging Software Reuse,” Stanford Law Rev., vol. 49, Jan. 1997, pp. 255–304. L. Graham, “Recent Patent Reform Legislation,” IEEE Software, vol. 17, no. 3, May/June 2000, pp. 101–103.

For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.

Posix

gigabit Ethernet

enhanced parallel ports

wireless token rings networks

FireWire

Computer Society members work together to define standards like IEEE 1003, 1394, 802, 1284, and many more.

HELP SHAPE FUTURE TECHNOLOGIES • JOIN A COMPUTER SOCIETY STANDARDS WORKING GROUP AT

computer.org/standards/ May/June 2002

IEEE SOFTWARE

83