,
Dr. Dobbs J O U R N A L
#379 DECEMBER 2005
SOFTWARE TOOLS FOR THE PROFESSIONAL PROGRAMMER http://www.ddj.com
DATABASE DEVELOPMENT Table Patterns & Matching Data Java & ORM Enterprise Application Logging Automating Database Functions Memory Management & Embedded Databases Summer of Code
WideCharacter Format String Vulnerabilities Amazon Web Services Inside the OCAP DVR Spec Ed Nisley
On the (Linux) Road Again
Grids & Data Mining XML-Binary Optimized Packaging Migrating a Mac Text Editor to Intel
Jerry Pournelle
Windows Vista: First Impressions
C O N T E N T S
DECEMBER 2005 VOLUME 30, ISSUE 12
FEATURES Table Patterns & Changing Data 14 by Todd Schraml
Historical data doesn’t need to clutter your database and slow its performance.
Rapid Data Access Tier Implementation 24 by John Cheng, Abdul Akbari, and Hong Rong
DBAG is a tool for automating the creation of domain-specific classes.
Month-Text Ordering 30 by David Wincelberg
A month field-type can be a useful addition to your database toolbox.
Object-Relational Mapping in Java with SimpleORM 34 by Martin Snyder and Ted O’Connor
SimpleORM is a lightweight — yet powerful — object-relational mapping implementation.
Building Grid-Enabled Data-Mining Applications 41 by Alex Depoutovitch and Alex Wainstein
Computing grids let you use parallelization to tackle really big data-mining jobs.
The OCAP Digital Video Recorder Specification 49 by Linden deCarmo
The OCAP Digital Video Record specification defines an open API for building DVR applications.
XML-Binary Optimized Packaging 53 by Andrey Butov
XML-Binary Optimized Packaging provides a means for including binary data within XML documents.
A Mac Text Editor Migrates to Intel 56 by Tom Thompson
BBEdit, an industrial-strength Macintosh text editor, has been ported to the Intel platform. Here’s how.
Google’s Summer of Code: Part I 60 by DDJ Staff and Friends
Google’s Summer of Code resulted in thousands and thousands of lines of code. Here are some of the students who participated.
Wide-Character Format String Vulnerabilities 63 by Robert C. Seacord
Robert presents strategies for handling format string vulnerabilities in C.
Amazon Web Services 66
EDITORIAL 8 by Jonathan Erickson
by Ashish Muni and Justin Hansen
ScanZoom lets you use mobile camera phones to launch services by taking photos of barcodes.
Enterprise Application Logging 68 by Jim Mangione
Accurately gauging the health of IT assets across applications requires true enterprise-level logging.
EMBEDDED SYSTEMS PROGRAMMING Memory Management & Embedded Databases 72 by Andrei Gorine & Konstantin Knizhnik
Embedded and in-memory databases depend on the quality of their memory-management algorithms.
COLUMNS Programming Paradigms 76
Chaos Manor 82
by Michael Swaine Michael reminiscences about OOPSLA ’89 in New Orleans.
by Jerry Pournelle Jerry shares his first impressions of Windows Vista.
Embedded Space 79
Programmer’s Bookshelf 85
by Ed Nisley Ed attends the 7th Annual Linux Symposium to find out what’s up with Linux.
by Gregory V. Wilson Six books in seven paragraphs! Greg’s up to it.
http://www.ddj.com
FORUM
Dr. Dobb’s Journal, December 2005
DR. ECCO’S OMNIHEURIST CORNER 10 by Dennis E. Shasha NEWS & VIEWS 12 by DDJ Staff PRAGMATIC EXCEPTIONS 18 by Benjamin Booth OF INTEREST 87 by DDJ Staff SWAINE’S FLAMES 88 by Michael Swaine
NEXT MONTH: Join us in January for DDJ’s 30th anniversary celebration. We also examine a bevy of programming languages.
3
D R .
D O B B ’ S
O N L I N E
C O N T E N T S Online Exclusives
The Perl Journal
http://www.ddj.com/exclusives/
http://www.tpj.com/
Visual Basic 9.0: Looking Forward
Improving Template::Extract
Scott Swigart talks with Microsoft’s Visual Basic team about some of the changes coming in VB 9.0.
Tightening a few loose screws in a module with lots of potential.
Are Standards Enough for Web-Services Security? The set of web-services standards seems to grow by the day. But if a web-services implementation supports all of these standards, is it necessarily secure?
The News Show
Dobbscast Audio
Windows in the Palm of Your Hands
http://www.ddj.com/podcast/
Software Builds: Where it all Comes Together
http://thenewsshow.tv/
Will smartphones take off as Microsoft and Palm hook up.
John Ousterhout, founder of Electric Cloud and creator of the Tcl scripting language, discusses the challenges in performing efficient software builds.
.NET Upgrade Metrics and the Migration Assessment Tool The new .NET Migration Assessment Tool provides stats across multiple projects to help measure upgrade costs.
Windows/.NET http://devnet.developerpipeline.com/windows/
Getting Runtime Info from the CLR You can update the Trace class to dump out some useful diagnostics about where the Trace object is being used
Dotnetjunkies http://www.dotnetjunkies.com/
Microsoft Enterprise Library 2005 The Enterprise Library is a collection of Application Blocks released by the Patterns and Practices group within Microsoft.
BYTE.com http://www.byte.com/
Apache WSRF, Pubscribe, and Muse Ian Springer explains how to use Apache’s toolkits for implementing web services based on WSRF, WSN, and WSDM.
The C/C++ Users Journal http://www.cuj.com
CUJ Experts Forum Bjorn Karlsson continues his series on the Algorithms Library of the C++ Standard with a discussion of three related algorithms —search, search_n, and find_end.
RESOURCE CENTER As a service to our readers, source code, related files, and author guidelines are available at http://www.ddj.com/. Letters to the editor, article proposals and submissions, and inquiries should be sent to
[email protected]. For subscription questions, call 800-456-1215 (U.S. or Canada). For all other countries, call 902563-4753 or fax 902-563-4807. E-mail subscription questions to
[email protected], or write to Dr. Dobb’s Journal, P.O. Box 56188, Boulder, CO 80322-6188. If you want to change the information you receive from CMP and others about products and services, go to http://www.cmp.com/ feedback/permission.html or contact Customer Service at Dr. Dobb’s Journal, P.O. Box 56188, Boulder, CO 80322-6188. Back issues may be purchased prepaid for $9.00 per copy (which includes shipping and handling). For issue availability, send e-mail to
[email protected], fax to 785-838-7566, or call 800-444-4881 (U.S. and Canada) or 785838-7500 (all other countries). Please send payment to Dr. Dobb’s Journal, 4601 West 6th Street, Suite B, Lawrence, KS 66049-4189. Digital versions of back issues and individual articles can be purchased electronically at http://www.ddj.com/.
WEB SITE A C C O U N T A C T I VA T I O N Dr. Dobb’s Journal subscriptions include full access to the CMP Developer Network web sites. To activate your account, register at http://www.ddj.com/registration/ using the web ALL ACCESS subscriber code located on your mailing label.
DR. DOBB’S JOURNAL (ISSN 1044-789X) is published monthly by CMP Media LLC., 600 Harrison Street, San Francisco, CA 94017; 415-947-6000. Periodicals Postage Paid at San Francisco and at additional mailing offices. SUBSCRIPTION: $34.95 for 1 year; $69.90 for 2 years. International orders must be prepaid. Payment may be made via Mastercard, Visa, or American Express; or via U.S. funds drawn on a U.S. bank. Canada and Mexico: $45.00 per year. All other foreign: $70.00 per year. U.K. subscribers contact Jill Sutcliffe at Parkway Gordon 01-49-1875-386. POSTMASTER: Send address changes to Dr. Dobb’s Journal, P.O. Box 56188, Boulder, CO 80328-6188. Registered for GST as CMP Media LLC, GST #13288078, Customer #2116057, Agreement #40011901. INTERNATIONAL NEWSSTAND DISTRIBUTOR: Source Interlink International, 27500 Riverview Center Blvd., Suite 400, Bonita Springs, FL 34134, 239-949-4450. Entire contents © 2005 CMP Media LLC. Dr. Dobb’s Journal® is a registered trademark of CMP Media LLC. All rights reserved.
4
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
,
Dr.Dobbs J O U R N A L
PUBLISHER Michael Goodman
SOFTWARE TOOLS FOR THE PROFESSIONAL PROGRAMMER
EDITOR-IN-CHIEF Jonathan Erickson
EDITORIAL MANAGING EDITOR Deirdre Blake SENIOR PRODUCTION EDITOR Monica E. Berg ASSOCIATE EDITOR Della Wyser COPY EDITOR Amy Stephens ART DIRECTOR Margaret A. Anderson SENIOR CONTRIBUTING EDITOR Al Stevens CONTRIBUTING EDITORS Bruce Schneier, Ray Duncan, Jack Woehr, Jon Bentley, Tim Kientzle, Gregory V. Wilson, Mark Nelson, Ed Nisley, Jerry Pournelle, Dennis E. Shasha EDITOR-AT-LARGE Michael Swaine PRODUCTION MANAGER Stephanie Fung INTERNET OPERATIONS DIRECTOR Michael Calderon SENIOR WEB DEVELOPER Steve Goyette WEBMASTERS Sean Coady, Joe Lucca AUDIENCE DEVELOPMENT AUDIENCE DEVELOPMENT DIRECTOR Kevin Regan AUDIENCE DEVELOPMENT MANAGER Karina Medina AUDIENCE DEVELOPMENT ASSISTANT MANAGER Shomari Hines AUDIENCE DEVELOPMENT ASSISTANT Melani Benedetto-Valente MARKETING/ADVERTISING ASSOCIATE PUBLISHER Will Wise SENIOR MANAGERS, MEDIA PROGRAMS see page 86 Pauline Beall, Michael Beasley, Cassandra Clark, Ron Cordek, Mike Kelleher, Andrew Mintz MARKETING DIRECTOR Jessica Marty SENIOR ART DIRECTOR OF MARKETING Carey Perez DR. DOBB’S JOURNAL 2800 Campus Drive, San Mateo, CA 94403 650-513-4300. http://www.ddj.com/ CMP MEDIA LLC Steve Weitzner President and CEO John Day Executive Vice President and CFO Jeff Patterson Executive Vice President, Corporate Sales & Marketing Leah Landro Executive Vice President, Human Resources Mike Mikos Chief Information Officer Bill Amstutz Senior Vice President, Operations Sandra Grayson Senior Vice President and General Counsel Alexandra Raine Senior Vice President, Communications Kate Spellman Senior Vice President, Corporate Marketing Mike Azzara Vice President, Group Director of Internet Business Robert Faletra President, Channel Group Tony Keefe President, CMP Entertainment Media Vicki Masseria President, CMP Healthcare Media Philip Chapnick Vice President, Group Publisher Applied Technologies Paul Miller Vice President, Group Publisher Electronics Fritz Nelson Vice President, Group Publisher Network Computing Enterprise Architecture Group Peter Westerman Vice President, Group Publisher Software Development Media Joseph Braue Vice President, Director of Custom Integrated Marketing Solutions Shannon Aronson Corporate Director, Audience Development Michael Zane Corporate Director, Audience Development Marie Myers Corporate Director, Publishing Services
American Buisness Press
6
Dr. Dobb’s Journal, December 2005
Printed in the USA
http://www.ddj.com
EDITORIAL
Winners and Losers
G
oogle’s Summer of Code was the cat’s pajamas. Not only did it generate thousands of lines of open-source code in a variety of programming languages for a bunch of operating systems, but it paid well, too. Student developers who were selected to participate — and there were more than 400 of them — received $4500 each for coding to their heart’s delight over the summer. But money wasn’t the only motivation, at least from Google’s perspective. According to promotions the company ran last spring, the goal was “to get the brightest minds on campus contributing code to open-source initiatives and inventing new open-source programs.” Still, no one turned down the loose change. After all, when it comes to summer jobs, buying coffee at Starbucks is a lot better than serving it. Led by Chris DiBona, Google’s open-source programs manager, and Greg Stein, a Google engineering manager, the Summer of Code team proceeded to ally student programmers with mentors (DDJ contributing editor Greg Wilson was one mentor) and mentoring organizations. The mentors and mentoring organizations came up with project ideas, acted as sounding boards, and monitored student progress. The mentoring organization also played a part in determining the platform (come on, you really didn’t think that, say, FreeBSD.org would sponsor a Windows or Mac OS X project) and licensing scheme for the source code (as one of the mentoring organizations, Google required BSD, LGPL, or GPL). The choice of programming language was left to the student. Mentoring organizations included the Apache Software Foundation, Blender, Drupal, FreeBSD, The Gnome Foundation, Internet2, the Jabber Software Foundation, KDE, the Mono Project, NetBSD, OpenOffice.org, the Perl Foundation, Portland State University, the Python Software Foundation, the Subversion Project, XWiki, and Google, among others. To acknowledge the efforts and contributions of students who participated in the Summer of Code, we will be presenting over the coming months profiles of many of the participants and their projects. We start off this month with a brief interview with Chris DiBona, who provides some background on the Summer of Code, then let the students tell you about their projects in their own words (see page 60). One thing you’ll notice right away is that the Summer of Code was truly an international effort. This month’s students, for instance, attend schools in Sri Lanka, New York City, Croatia, and Missouri. Please join me in congratulating the students for their fine work and Google for sponsoring the initiative. From what I can tell, the Summer of Code was an all-around winner — the mentoring organizations got a lot of great code to support their projects, the programming community was reminded about the importance of creativity and cooperation, and the students can now afford something other than Ramen noodles for dinner. Okay, that takes care of the winners. Now for the losers. Several years ago, DDJ published an article about the baggage system at the Denver International Airport (see “Simulating the Denver Airport Automated Baggage System,” by John Swartz, DDJ, January 1997). It didn’t take a massively parallel grid computer and army of consultants to figure out whether the $200 million 17-mile-long Rube Goldberg system of tracks and carts would work. (In fact, all it took was a run-of-the-mill PC and XLisp, David Betz’s freely available Lisp implementation.) What John Swartz’s simulation confirmed was that the automated baggage system wasn’t feasible. Nearly a decade and millions of dollars later, it’s official — the automated baggage system won’t work and has been abandoned. United Airlines, the only airline still using the system, finally just said “no” to automation, opting for human baggage handlers and saving $1 million a month in maintenance costs in the process. The system, designed by BAE Automated Systems (which closed its doors years ago), was a paragon to technology. It originally called for 300 OS/2-based 486 PCs distributed in eight control rooms, databases running on fault-tolerant servers, high-speed fiberoptic Ethernet, 14 million feet of wiring, 56 laser arrays, 400 frequency readers, 22 miles of track, six miles of conveyor belts, 3100 standard carts, 450 oversized carts, 10,000 motors, and 92 PLCs to control motors and track switches. And, as Daniel Stearns of California Polytechnic State University notes, the system accurately illustrated the principle of object-oriented design by sending “messages to objects [the carts], which respond by returning other objects [baggage and empty carts] to the sender.” But it didn’t work. Granted, offering advice at this point is like closing the barn door after the baggage cart has escaped. Still, you have to wonder whether a cadre of creative, energetic, and dedicated students looking to avoid a summer of saying, “Would you like fries with that?” could have done at least as good a job as those losers who planned (and profited from) the Denver Airport fiasco. Google and its Summer of Code just may be on to something, and I’m already looking forward to the Summer of Code 2006.
Jonathan Erickson editor-in-chief
[email protected] 8
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
DR. ECCO’S OMNIHEURIST CORNER
The Box Chip Game Dennis E. Shasha
D
r. Ecco seemed to be in a mood to muse on a cool fall day in his McDougal Street apartment. “Sometimes a problem comes along for which a seemingly insignificant twist completely changes its character,” he said. “This holds particularly in games of chance. You remember our friend Jimmy Casino, don’t you? Well, he invented a new game. Want to hear it?” “I’d be delighted,” I said thinking fondly of the baby-faced huckster who had visited us a few months earlier. Ecco began: “You have some even number n of chips, each having a different color. You put one in each box. Your adversary rearranges the boxes behind a curtain. You then rearrange them further once they’re on your side of the curtain. (At the end of the game, you will be able to verify that there is a chip of each color in the boxes, so you can be sure that the adversary does no more than rearrange the boxes.) The net effect is that you can assume the rearrangement is entirely random. “For each color, you guess n/2 boxes where that color might be. You make all Dennis, a professor of computer science at New York University, is the author of four puzzle books: The Puzzling Adventures of Dr. Ecco (Dover, 1998); Codes, Puzzles, and Conspiracy (Freeman 1992, reprinted by Dover in 2004 as Dr. Ecco: Mathematical Detective); and recently Dr. Ecco’s Cyberpuzzles (W.W. Norton, 2002); and Puzzling Adventures (W.W. Norton, 2005). With Philippe Bonnet, he has written Database Tuning: Principles, Experiments, and Troubleshooting Techniques (2002, Morgan Kaufmann). With Cathy Lazere, he wrote Out of Their Minds: The Lives and Discoveries of 15 Great Computer Scientists (1995, Copernicus/Springer). He can be contacted at
[email protected]. 10
guesses about all colors before any box is opened. If you are correct about every color, you win. Otherwise, you lose. It might seem that your chances of winning are (1/2)n, but you can do much better. “Warm-Up: Suppose there are two chips — blue and red — and two boxes. So each guess is about just one box. Which guesses can you make so your chances of being right in both guesses is 1/2?” “That one is easy,” I said. “Guess red in the left box and blue in the right box. You’ll either be both correct or both wrong.” “Right,” said Ecco. “Now let’s say there are four chips (n= 4) whose colors are black, white, red, and green. You are allowed to guess two boxes for each chip. Number the boxes 1, 2, 3, 4. To start with, your guesses have to be a simple list of box numbers for each color. For example: Black guess: 1, White guess: 2, Red guess: 1, Green guess: 3,
2 4 3 4
“1. For four chips and two guesses per color, can you figure out a way to win at least 1/6 of the time assuming all rearrangements are equally likely? (The example just given might not do this.) “2. What if there are six chips and three guesses per color? “3. How does this probability generalize for n chips (n even) and n/2 guesses per color? “Now here comes the small twist. Under the old rules, for each color, you provide n/2 boxes as guesses. Under the new rules, for each color, you provide an initial guess and a procedure for determining each next guess based on the actual color found by the previous guess. “Example for black in the case of four colors: Dr. Dobb’s Journal, December 2005
Start with box 1; case result of first step is Black, then win Red, then try 2 White then try 3 Green then 4
“For convenience, we will abbreviate this as follows: black guess: 1; black -> win, red -> 2, white -> 3, green -> 4 “All initial guesses and procedures are provided before any checking occurs and the procedures may not share information. If every color is found in n/2 or fewer guesses, you win. Make sense?” “I think I understand,” I said. “We might conceptualize the situation as follows: Each color is represented by an agent who follows a procedure like the one above, possibly different procedures for different agents. A given agent enters the room with the boxes, looks inside the boxes as dictated by his procedure but doesn’t otherwise disturb the boxes, then leaves the scene without being able to communicate with any other agents.” “Exactly,” said Ecco. “Continuing with your scenario: A judge determines whether that ‘procedural agent’ has won or not. If all agents win, then you (the person betting) win. Otherwise, the house wins. It may seem surprising but you can do a lot better under these new rules. “4. How well can you do in the case of four chips and two guesses under the procedural agent assumption? (Hint: You can win over 40 percent of the time.) “5. How well can you do if you have six chips and three guesses?” For the solution to last month’s puzzle, see page 78.
DDJ http://www.ddj.com
SECTION
A
MAIN NEWS
Dr. Dobb’s
News & Views
New Learning Algorithm Researchers at Cornell University and Tel Aviv University have devised a technique for scanning text in a variety of languages, then autonomously learning its complex syntax. This information can then be used to generate new and meaningful sentences. The Automatic Distillation of Structure (ADIOS) algorithm, which relies on a statistical method for pattern extraction and on structured generalization, discovers the patterns by repeatedly aligning sentences and looking for overlapping parts (http:// adios.tau.ac.il/). The technique has implications for speech recognition and naturallanguage engineering, genomics, and proteomics. The algorithm was developed by David Horn and Eytan Ruppin, professors at Tel Aviv University, with Ph.D. student Zach Solan.
Infrared Specs Released The IrSimple Protocol and Profile Specifications, released by the Infrared Data Association (http://www.irda.org/), were designed for fast, wireless communication between mobile devices and digital home appliances. Data rates up to 16 Mbps (VFIR) are currently available with 100 Mbps (UFIR) under development. For example, the specs will enable mobile devices such as camera phones to instantly connect and wirelessly transmit digital images to similarly enabled TVs, monitors, projectors, and photo kiosks. The organization claims that there are more than 750 million IrDAenabled devices in the global market today.
Optical Storage Spec Updated The Optical Storage Technology Association has released its Universal Disc Format (UDF) Rev. 2.60 filesystem specification (http://www.osta.org/specs/). A major addition in Rev. 2.60 is the addition of a new Pseudo OverWrite (POW) mechanism that supports sequential recording on new types of write-once discs and drives such as BDR (Blu-Ray Disc-Write Once). The special logical overwrite function of the POW mechanism that enables write-once media to behave more like a rewritable disc was developed in parallel by the Blu-Ray Disc Association (BDA). Rev. 2.60 with POW also increases disc compatibility between consumer video recorders and computer systems, and allows use of the Metadata File to locate metadata in a logically contiguous manner for increased efficiency. UDF is a filesystem first defined by OSTA 12
in 1997 to support transfer of Magneto-Optical discs and files between different computer systems. The specification, based on the ECMA-167/ISO 13346 Standard, is intended for developers planning to implement UDF to enable disc and file interchange among different operating systems.
GPL Update Underway The Free Software Foundation (FSF) has launched the GPL Version 3 Development and Publicity Project (http://www.fsf.org/ news/gplv3/), with the goal of updating the General Public License (GPL). The current version of the license was written in 1991. Among the issues expected to be addressed are distribution and licensing. Free Software Foundation members have been invited to participate in various committees, starting with advisory committees that will specify the objectives and parameters for participation in GPLv3. Guidelines are expected to be released by December 2005, with a draft ready in early 2006.
International Programming Challenge Held More than 70 countries were represented in this year’s International Olympiad on Informatics, an annual world computing championship contest for precollegiate students held this year in Nowy Sacz, Poland. The competition consisted of two five-hour programming sessions, each with three programming challenges. The four USA Computing Olympiad (http://www.usaco.org/) team members —Alex Schwendner, Eric Price, John Pardon, and Richard McCutchen — all won gold medals. Additionally, Eric Price received a special Grand Award for his perfect score of 600. Poland’s Filip Wolski was crowned as the World Champion. The USACO is sponsored by Equinix, USENIX, SANS, the ACM, and IBM.
NASA Honors Two Projects The National Science and Space Administration has announced that two of its teams will receive the agency’s Software of the Year Award. A team from NASA’s Goddard Space Flight Center was recognized for its Land Information System Software 4.0, while a team from the Jet Propulsion Laboratory was acknowledged for its Autonomous Sciencecraft Experiment (ASE) software. LIS (http://lis.gsfc.nasa.gov/) is a land surface modeling and data assimilation system that predicts water and energy cycles such as Dr. Dobb’s Journal, December 2005
DR. DOBB’S JOURNAL December 1, 2005
runoff, evaporation from plants and soil, and heat storage in the ground. ASE (http:// ase.jpl.nasa.gov/) makes it possible for autonomous spacecrafts to increase their science return by two orders of magnitude via onboard decision making. ASE is used for detecting and tracking environmental events on Earth, such as volcanic eruptions, floods, and wild fires.
CA Releases Software Patents Computer Associates International (http:// ca.com/) has released 14 of its patents to individuals and groups working on opensource software. In the process, CA joined IBM in encouraging an industry-wide “patent commons” in which patents are pledged royalty free to further innovation. The patents covered by CA’s move generally include: Application development and modeling that automates translation between programming languages; business intelligence and analytics; systems management and storage-management solutions that provide intelligent process controls to maximize system performance and storage utilization; and network management and security tools that enhance visualization of network traffic patterns and congestion, selectively capture and filter network traffic, and provide granular session-control capabilities.
IBM Launches Transition-to-Teaching Program To help address a nationwide shortage of math and science teachers, IBM has launched a Transition-to-Teaching program that enables as many as 100 U.S. employees to gain teacher certification. IBM will reimburse participants up to $15,000 for tuition and stipends while they student teach, as well as provide online mentoring and other support services in conjunction with partner colleges, universities, and school districts. According to the U.S. Department of Labor, more than 260,000 new math and science teachers will be needed by the 2008 –2009 school year. The IBM pilot will be operational in January 2006 in New York, North Carolina, and other locations where IBM has a significant population. Employees will need management approval and must fulfill requirements such as 10 years of service with IBM, a bachelor’s degree in math or science or a higher degree in a related field, and some experience teaching, tutoring, or volunteering in a school or other children’s program. http://www.ddj.com
Table Patterns and Changing Data Dealing with history in relational databases TODD SCHRAML
W
ith the financial reporting requirements of the SarbanesOxley Act, businesses have good reason to be concerned about the past. Enhancements to existing applications and changes that incorporate auditing needs and point-in-time recoverability in new application development have moved from back-burner wish-lists to mission-critical, immediate necessities. Beyond backups and logs at the DBMS level, there are numerous options when considering history-preserving data structure configurations. For instance, you can add start/stop timestamps or currency flags to tables in various fashions. Sometimes you can add dates to unique indexes. Best practice necessitates mapping historical needs to chosen historical data structures so that they work well together. But problems arise for both applications and users if historical data structures do not align with actual usage of the contained history. Historical data can easily clutter a database, ultimately slowing down performance, or forcing jumps through SQL-hoops to access the correct active subset of rows in a table. There are any number of permutations that address these problems. The approach I present here offers five structural archetypes that involve the fusion of time and data inside database tables. I also examine the nature of temporal data structure patterns, and provide guidelines for establishing a history-management strategy that can be leveraged across an organization. Regardless of the implemented data structure, a few basic auditsupporting attributes should apply to all tables. These attributes (serving as basic columns) encompass tracking a row’s creation, and timing a row’s last change. I recommend incorporating four standard columns on virtually every table. The four standard column titles would be some variation on “Row-Created Timestamp,” “Row-Creator ID,” “Row Last Updated Timestamp,” and “Row Last Updated ID.” The last two columns are unnecessary in situations warranting only inserts with no row updates. Thus, even on tables that only contain current data, the proper population of these four columns (along with a comprehensive set of database backups) should provide a considerable amount of point-in-time recoverability. The temporal patterns (or archetype structures) I examine here generally echo the standardized approaches used for handling time as found inside existing business-intelligence-related (BI) multidimensional data-management practices. A variant of these BI ritual practices applies these data-management practices across all kinds of database tables, especially tables used Todd is a data architect at Innovative Health Strategies. He can be reached at
[email protected]. 14
for supporting operational applications. Multidimensional practice (like that found in star schemas) simplifies the collection of data elements into two kinds of table structures. Items are grouped together into: • Fact tables that contain all numeric data intended for analysis. • Dimension tables that store the various text data items available for selecting, grouping, or ordering the fact statistics.
“Top priorities should include evolving a strategy that identifies the preferred historical archetypes” Pointers inside the fact table rows provide the proper link to dimension rows. In many respects, the dimension tables function as indexes into the associated fact tables. Within these dimensional structures, there are three standard update strategies employed for managing value changes over time. The dimension update process is generically called “Slowly Changing Dimension,” and these strategies are “Type 1,” “Type 2,” and “Type 3.” • Type 1 strategy retains only current values because the associated columns in a dimension table do not require the retention of history. • Type 2 strategy inserts a new row every time one of the column values changes, which always allows association of the crucial fact with the original value of dimension columns. • Type 3 strategy actually adds an additional column onto the dimension table, providing retention of both old and new values (Old_Customer_Name and New_Customer_Name). For generalized database usage, the value change management archetypes in this discussion include the: • Current-only table. • Functional-history table. • Row-level audit (or shadow) table. • Column-level audit table. • Ledger table.
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
(continued from page 14) Each of the structures can serve a unique kind of circumstance (as described in Table 1). Therefore, when building new applications, the database design process should include a table-bytable evaluation, identifying which type of temporal functions best suit the overall application needs. Moreover, these approaches are widely used today by developers everywhere. My focus here is on organizing these options into a framework that leads to a cohesive and organized development approach. Current Only Clearly, instances occur where you need only the data as it currently stands. For example, an order-taking system may require a table that manages the next available service date. As capacity is scheduled for use, that availability date moves ahead. Knowing what the next available service date was as of last week or last month may not be useful. The current-only table (Figure 1) simply offers an original version of any table that omits start or stop timestamping columns, and provides an important tool to keep in the toolchest. Remember that even under the eyes of the Sarbanes-Oxley compliance reaper, some data may still have no historical value. (However, it is likely that there is much less data categorized in this manner than was considered of little historical value previously.) Furthermore, circumstances may dictate using a current-only table for performance, while using an additional table-type for duplicating the historical aspects of the data. It may be a bit obvious, but the current-only approach is indeed equivalent to the multidimensional Type 1 strategy. Functional History When most people think of “keeping history” in their database, some variation of the functional-history table is often what comes to mind (Figure 1). A functional-history table is built by adding an activity start date and an activity stop date to a basic table structure. The row retains historic values frozen in time. When a value changes, then the original row has its activity stop date populated with the current date, and a new row, using the current date as the start date and the new column values, is inRequirement
Current Only
Application only wishes to see current active values for all functional tasks against data store.
serted. Logically, this is the equivalent of the previously mentioned dimension Type 2 updating approach. Fundamentally, this functional-history table alteration does revise the meaning of the table involved. A table originally worked as an “Employee” or “Order” table now becomes an “Employee Activity” or “Order History” table. This meaning adjustment occurs because a single item (either a single employee or a single order, per the given examples) exists in multiple rows after making this change. Consequently, effective use of the functional-history table requires making another change or two. Previously, a unique index likely existed on the item identifier (the “Employee ID” or the “Order Number”). Because any single Employee ID value may be duplicated across multiple rows as changes unfold over time, that index needs to include the activity start-date column. With this start-date addition, the index remains unique. If you manage data models, the functional-history table, in all its time-laden glory, should exist within the conceptual data model because the functional dependencies and joins should expect a time component. Additionally, while many application tasks read this functional-history table, some need only the current values. It becomes tedious rewriting code over and over in search of a maximum start-date value. Likewise, using a test for NULLs in a stop-date column in order to determine currency can be an issue because NULLs and indexing sometimes conflict with each other. Therefore, a fairly common practice when dealing with functional-history tables involves adding a column that serves as an “Active Indicator.” A “Live or Dead” flag provides a simple test to retrieve only current rows. As a DBMS managed view can be defined using the flag, even this simple test remains hidden from the application code. Row-Level Audit The similarity of the row-level audit table to the functional-history table results from retaining values of each change event that impacts table content (see Figure 2). A distinct difference from the functional-history table exists because the row-level audit table is not meant to change the base table, but serve as an add-on structure so that you now have two tables — the original and the
Can Fit Criteria Functional History Row-Level Audit
Column-Level Audit Ledger
X
Normal application tasks will require access to data valid at specific and varying points-in-time.
X
Application only wishes to see current values, but auditors need to be able to see every change.
X
Application only wishes to see current values, but users want to browse through changes made to specific columns.
X
X
X
Critical data is numeric and must always be able to know about every change impacting figures.
X
Normal application tasks will require access to data valid at specific and varying points-in-time for a subset of data. Auditors will need to be able to see every change made.
X
X
Normal application tasks will require access to data valid at specific and varying points-in-time for a subset of data. Users want to browse through changes made to specific columns. Auditors will need to be able to see every change made.
X
X
X
Table 1: Considerations in choosing a structure. 16
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
(continued from page 16) row-level audit version keeping history! The row-level audit table is a virtual copy of the base table with an addition of two columns. One column contains the timestamp of the change event, while the second column identifies the basic nature of that change event— for example, as an INSERT, UPDATE, or DELETE. Any change to the base table also inserts a row into the row-level audit table. This duplication of content and effort suggests the preference to name the row-level audit table a shadow table. In a sense, this shadow table is beneath the surface of the main application functionality, making it a hypofunctional-history table more than a functionalhistory table. By using a shadow structure, a current-only table can remain as part of an application, and its link to a shadow version retains change history in case the application is audited. Column-Level Audit Another potentially useful add-on table is the column-level audit table. While ostensibly a simple variation of the previous
Pragmatic Exceptions . . . . .
Figure 1: Current-only and functional-history structure.
18
Tip #4: Make Only the Actionable Obvious A general rule for showing obvious exception messages (such as in a dialog) should be: Show the message only if there is an immediate, reasonable course of action users can take. If none can be imagined, then either the exception can be quietly logged in the background or it’s serious enough to crash the application now, before doing any further damage. An example of an actionable message would be: “The file you were editing was updated; load the new version?” File changes to the currently edited file are exceptional conditions that, internally, could very well generate exceptions. In such a case, showing users what happened and offering some course of action is the best approach. But what about showing a dialog (or console message) just before the application crashes that points to the the log file? Go for it. This still fits the rule since they can take action by examining the log. —Benjamin Booth http://www.benjaminbooth.com/
row-level audit, it differs from the shadow table because everything has gone vertical. The row has been placed up-anddown rather than side-to-side (see Figure 2). By up-ending things in this vertical manner, the table now must include descriptive metadata. In fact, this audit table looks nothing like the original table being tracked. Structurally, the column-level audit table contains the key from the original table, a timestamp of the value change event, the name of the column with changing data, the old/original value, and lastly, the incoming new value. The type of change event may be implied by the data (a nulled old value column implying an INSERT, or a nulled new value implying a deletion), or these actions may be made explicit within another column of the table. This structure assumes that the application tracks exactly which columns are changing and only inserts rows for those columns with altered values. Including arbitrary inserts of all columns for any change, removes any usefulness from this approach because the table explodes with an overabundance of details that must be sifted to find a true change. By keeping both old and new values in this manner on a single row, the column-level audit table provides the functionality implied by a Type 3 Slowly Changing Dimension. Applications often make use of column-level audit tables in cases that include a detailed application requirement, thus allowing users the capability to browse specific data changes easily. Under these requirements-driven circumstances, change tracking is usually limited to specific column and specific change events. The column-level audit table’s existing format allows for fairly meaningful row browsing with less additional formatting necessary for presentation. Ledger The ledger table is a specialty table. This history archetype is unlike any of the “Slowly Changing Dimension” approaches. The distinction for this unlikeness occurs because the ledger table is treated like the fact table in multidimensional designs. In the past, the ledger table structure was useful only in rare and specific
Figure 2: Row-level and column-level audit structures.
Figure 3: Ledger table.
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
(continued from page 18) circumstances, such as storing data for an actual general ledger application. Although it still fills those same requirements today, the ledger structure’s usefulness is no longer quite so rare. As implied by the ledger moniker, a ledger table acts like an accounting book with the application of credits and debits (see Figure 3). With proper use of the ledger archetype, only inserts occur on this table. The ledger table is a personal favorite, often referred to as a “sticky” table; like a fly strip, once data lands here, it is meant to stay there forever. The ledger table can require a significant amount of work. For this technique to function properly, all numbers must “balance” when summed. This means that when items change, the changes must be evaluated before allowed for insertion to the ledger table. If a previous column entry of +5 must be “updated” to a +6 value, rather than updating the +5 to a +6, either a “net change” or “gross change” approach must be used to insert new rows into the ledger table. Using a net change approach, a row is inserted for a value of +1 (see Figure 3). The previous +5 and the new +1 will sum up to the correct +6 value. Under the gross change tactic, two rows are inserted for the single change. First, a row for – 5 is inserted, which logically reverses the previous +5 row. Next, a row for the +6 is inserted (see Figure 4). The ledger table, by its nature, is transactional. Circumstances of spotty or incomplete transactional pictures in the incoming data content mean that greater work must be done by the process creating the ledger table inserts. Putting It All Together Any specific application requires incorporating several of the structures described here; see Listing One. Even a single table may take advantage of a combination of approaches. Actually, the row-level or column-level audit functions best in tandem with either a current-only or functional-history structure. Other useful configurations may not be so obvious. For example, a functional-history table may be implemented only partially by using a subset of columns for triggering new row insertion. (For example, a change in the status code causes an update to the stop date and the insert of a new row; but an update of the text inside a comment column may not cause any response at all on this functional-history table.) Because of this partiality of the functional-history structure, perhaps a shadow table is implemented to catch all changes. In this fashion, changes that do not drive a new row insertion are not lost. While this double-table approach does increase overall application complexity, it may form a valid choice under certain circumstances. History concerns do not have a single one-size-fits-all solution. As the various detailed archetypes demonstrate, a great deal of flexibility exists in describing the nature of a table’s relationship to change over time. Current- only tables may contain noncritical data items or may be used in conjunction with other archetypes, such as row-level audit, to cover time-variant value changes. Applications often use Listing One /*
TABLE: Employee_Current_Only
functional-history tables so that these tables are probably the most typical of our historical data archetypes. Row-level audit structures can allow for a current-only operational table to remain smaller and more quickly responsive to queries. Column-level audit tables offer value for users who wish to browse precise changes within the application itself. And lastly, under the right set of circumstances, ledger tables can maintain an explicit track of fully comprehensive transactional change. Alternately, overlooking the evaluation of these options can leave systemic blind spots. Such a case could happen if employing a functional-history that only inserts rows on a limited set of changes to handle one user requirement, but without duly considering a different requirement that all changes be tracked for auditing. Therefore, a hole that could have been filled by adding in a row-level audit table was missed. On the other hand, if you created a column-level audit table because some changes needed to be browsed interactively online, but then simply continued to use that one column-level audit structure for all other change tracking, a monster is born. The single column-level audit table would quickly become the largest table in the application (row-wise) and create a hazardous performance bottleneck. Ideally, within a development group or organization, top priorities should include evolving a strategy that identifies the preferred historical archetypes and deciding a preference of the history archetype options for developers to employ. Documentation can be as simple as describing the history archetypes, or their variations germane to a specific shop’s environment, and enumerating appropriate conditions suitable for the use of each archetype. Providing such a carefully thought out course of action often expedites application development and minimizes history retention implementation problems. When development practices incorporate detailed plans for establishing and maintaining history, these plans help achieve a greater level of consistency and reuse across all of an organization’s applications. DDJ ELSE PRINT '<<< FAILED CREATING TABLE Employee_Current_Only >>>' go
*/
CREATE TABLE Employee_Current_Only( Employee_ID int NOT NULL, Employee_Name varchar(100) NOT NULL, Employment_Status varchar(10) NOT NULL, Create_Dt datetime NOT NULL, Create_ID char(10) NOT NULL, Altered_Dt datetime NOT NULL, Altered_ID char(10) NOT NULL, CONSTRAINT PK_Employee_Current_Only PRIMARY KEY CLUSTERED (Employee_ID) ) go IF OBJECT_ID('Employee_Current_Only') IS NOT NULL PRINT '<<< CREATED TABLE Employee_Current_Only >>>'
20
Figure 4: Ledger structure using a gross change approach.
/* TABLE: Employee_Functional_History */ CREATE TABLE Employee_Functional_History( Employee_ID int NOT NULL, Start_Dt datetime NOT NULL, Employee_Name varchar(100) NOT NULL, Employment_Status varchar(10) NOT NULL, Create_Dt datetime NOT NULL, Create_ID char(10) NOT NULL, Altered_Dt datetime NOT NULL, Altered_ID char(10) NOT NULL, Active_Ind char(1) NOT NULL, Stop_Dt datetime NULL, CONSTRAINT PK_Employee_Functional_History PRIMARY
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
KEY CLUSTERED (Employee_ID, Start_Dt) ) go IF OBJECT_ID('Employee_Functional_History') IS NOT NULL PRINT '<<< CREATED TABLE Employee_Functional_History >>>' ELSE PRINT '<<< FAILED CREATING TABLE Employee_Functional_History >>>' go /*
TABLE: Employee_Row_Level_Audit
*/
CREATE TABLE Employee_Row_Level_Audit( Employee_ID int NOT NULL, Start_Dt datetime NOT NULL, Employee_Name varchar(100) NOT NULL, Employment_Status varchar(10) NOT NULL, Create_Dt datetime NOT NULL, Create_ID char(10) NOT NULL, Altered_Dt datetime NOT NULL, Altered_ID char(10) NOT NULL, Active_Ind char(1) NOT NULL, Action_Code char(10) NOT NULL, CONSTRAINT PK_Employee_Row_Level_Audit PRIMARY KEY CLUSTERED (Employee_ID, Start_Dt) ) go /* TABLE: Employee_Column_Level_Audit_Option_1
*/
CREATE TABLE Employee_Column_Level_Audit_Option_1( Employee_ID int NOT NULL, Event_Timestamp datetime NOT NULL, Column_Name varchar(80) NOT NULL, Old_Text varchar(100) NULL, New_Text varchar(100) NULL, Old_Number decimal(10, 2) NULL, New_Number decimal(10, 2) NULL, Old_Date datetime NULL, New_Date datetime NULL, Event_ID char(10) NOT NULL, CONSTRAINT PK_Employee_Column_Level_Audit_Option_1 PRIMARY KEY CLUSTERED (Employee_ID, Event_Timestamp) ) go
CREATE TABLE Employee_Column_Level_Audit_Option_2( Employee_ID int NOT NULL, Event_Timestamp datetime NOT NULL, Column_Name varchar(80) NOT NULL, Old_Text varchar(100) NULL, New_Text varchar(100) NULL, Old_Number varchar(30) NOT NULL, Event_ID char(10) NOT NULL, CONSTRAINT PK_Employee_Column_Level_Audit_Option_2 PRIMARY KEY CLUSTERED (Employee_ID, Event_Timestamp) ) go IF OBJECT_ID('Employee_Column_Level_Audit_Option_2') IS NOT NULL PRINT '<<< CREATED TABLE Employee_Column_Level_Audit_Option_2 >>>' ELSE PRINT '<<< FAILED CREATING TABLE Employee_Column_Level_Audit_Option_2 >>>' go /*
TABLE: Employee_Paycheck_Ledger
*/
CREATE TABLE Employee_Paycheck_Ledger( Employee_Paycheck_Event_Key uniqueidentifier NOT NULL, Employee_ID int NOT NULL, Pay_Date datetime NOT NULL, Pay_Amount decimal(10, 2) NOT NULL, Event_Type char(10) NOT NULL, Created_Dt datetime NOT NULL, Created_ID char(10) NOT NULL, CONSTRAINT PK_Employee_Paycheck_Ledger PRIMARY KEY CLUSTERED (Employee_Paycheck_Event_Key) ) go IF OBJECT_ID('Employee_Paycheck_Ledger') IS NOT NULL PRINT '<<< CREATED TABLE Employee_Paycheck_Ledger >>>' ELSE PRINT '<<< FAILED CREATING TABLE Employee_Paycheck_Ledger >>>' go IF OBJECT_ID('Employee_Row_Level_Audit') IS NOT NULL PRINT '<<< CREATED TABLE Employee_Row_Level_Audit >>>' ELSE PRINT '<<< FAILED CREATING TABLE Employee_Row_Level_Audit >>>' go
IF OBJECT_ID('Employee_Column_Level_Audit_Option_1') IS NOT NULL PRINT '<<< CREATED TABLE Employee_Column_Level_Audit_Option_1 >>>' ELSE PRINT '<<< FAILED CREATING TABLE Employee_Column_Level_Audit_Option_1 >>>' go /*
TABLE: Employee_Column_Level_Audit_Option_2
*/
DDJ
Rapid Data Access Tier Implementation Automating the creation of domain-specific classes JOHN CHENG, ABDUL AKBARI, AND HONG RONG
I
n today’s enterprise software applications, the database plays an important role. Hence, the database access tier becomes a vital component. Luckily there are many libraries, such as ODBC, JDBC, ADO, and the like, that simplify implementation. Nonetheless, finding suitable design patterns and implementing them involves the tedious task of writing domain-specific access classes. This is still the responsibility of developers. Moreover, a considerable amount of time needs to be spent maintaining this layer when the schema is not static, but continues to evolve, especially when the database schema contains a large number of tables. In this article, we present techniques for automating this process. In doing so, we introduce a lightweight tool that you can use to generate the access layer from a template that can be customized to the various access libraries. Data Entity and Table Relational Data Mapping A database access tier needs to manage two tasks: • Handle the encapsulation of the relational data stored in a database. John is a senior software engineer and Abdul is a technical manager at Sabre Airline Solutions. Hong is an independent consultant. They can be contacted at
[email protected], abdul_
[email protected], and hongrong1@ comcast.net, respectively. 24
• Implement the four basic operations: insert, select, update, and delete. (These operations are sometimes also known as “CRUD,” short for Create, Read, Update, and Delete.) It is generally preferable to implement these two tasks with different objects — one to encapsulate data, and the other to handle the access logic. The one that encapsulates data is usually called “Data Entity” (DE) or “Business Entity.” The one that implements access logic is normally called “Data-Access Logic Component” (DALC) or “Data-Access Object.” Another reason for separating them is when the DE needs to be moved from one tier to another in multitiered systems. Separate tiers can directly use the DE, thus avoiding conversion from one format to another. Typically, the DALC is directly dependent on the language, library, and database connection management and does not lend itself well when transferred from one tier to another. For single-tier apps, combining the two components has advantages. The biggest advantage of doing this is that it is simpler. Within the same component, it is easier to understand and quick to implement. It cuts the number of objects by half, and this number may be large for a typical medium- to large-size database. Actually, many applications choose to do so. Data entities can also be categorized according to how they are mapped to a table or tables. A data entity can map to a single table with data members corresponding to the columns, or it can map to a join of tables with data members corresponding to columns across the tables. Note that here you care more about how the columns are mapped to the data members, not how instances of each data entity map to the rows of a table. For the entities that directly map to a table, all CRUD operations can be done easily with the help of data-access logic components. For the entities that map to joints, in many cases it’s not possible or desired to perform the operations other than the select Dr. Dobb’s Journal, December 2005
(read). When considering the member-tocolumn mapping, the database metadata that need to be considered include table name, column name, column ordinal, column type, and column constraints (key, size, nullable, and so on). The code side information that needs to be considered includes: programming language, data type, and member access permission.
“Data entities can also be categorized according to how they are mapped to a table or tables”
Mapping the relationships between tables is another important point that must be considered. On the database side there are one-to-many, many-to-one, and manyto-many relationships. These relationships are established through foreign keys and many-to-many tables. On the code side, these relationships can be represented as parent-to-child, child-to-parent, and peerto-peer. The business logic tier can use these relationships for navigation from one entity to another. Problems and Benefits of Implementing the Data-Access Tier Implementation of the whole data-access tier, including data entities and data-access logic components, is tedious and time consuming. Each substantial table may require one or more classes and interfaces. For a database that contains hundreds or more http://www.ddj.com
(continued from page 24) tables, the task of coding is huge. Furthermore, if the database schema continues to evolve, the maintenance on the code side is also considerable. To simplify the work, developers have attempted to get help from code generators that allow tables and columns in the schema to be mapped to classes and members of the generated code (see Listing One). Code generation is a big leap in reducing manual work. It helps make the whole data-access tier code more consistent and predicable. Any improvements worked out on one database-access class can quickly and easily be applied to the other classes by means of code generation. Any mismatch between a database schema and the application that uses this schema can be detected by a compiler. For example, if a column is modified or removed, the generated code will reflect this change on a class member or function. Thus, any client code that uses this class member or function will be forced to make the necessary changes. It leaves no unpleasant surprises at runtime. This ease of propagating changes to the code, along with the error detection, forms a seamless link between the database schema and the application code. Several code generators and persistence frame-
works are available, both commercially and through open source. Some advanced code-generation tools provide another important feature — they let SQL statements be constructed with the generated information at runtime, which helps avoid schema mismatch (Listing Two). For instance, assume you collected the table and column information, such as type, name, size, and ordinal, into some static class members. Then, if you have some assistant objects that can utilize this collected information to build the SQL queries for data-access logic, you can say that your SQL queries are also “bonded” to the database schema. When writing code to generate the dataaccess tier, it requires that you first know what you want; that is, the look and behavior of the code. Then, it requires that you know how to get what you want — that is, how to write the generator. There are some problems with this strategy. For one thing, the generator is specific to one project. For another, the logic of the target code and the logic of the generator itself are often mixed together. It requires you to devote more time to coding and testing, and adhere to a more disciplined practice. Here we introduce a tool called “Database Access Generator” (DBAG), developed by Hong Rong. (This code gen-
erator is copyrighted by Hong Rong with the Open-Source Software Alliance License [OSSAL]. The full license content is captured in the license.txt file.) The difference between DBAG and similar tools is that it makes extensive use of code-generation templates, which can be easily expanded. They also adapt easily to the various flavors of an access driver within a particular access technology (for instance, the ADO.NET driver from Microsoft versus that from Oracle). DBAG predefines the common database schema information as keywords. These keywords can be used in a customer template, which is then read by the generator to produce code. The primary goal for DBAG is to create a lightweight persistence framework through code generation. Some of the features of this tool are: • Code generation is fully templatized. • Available under open source. • Can handle complex situations such as joins and relationships. • Strong type checking. • Schema matching for class, class members, and SQL queries. • Null handling. • Tested with Oracle and SQLServer, but not limited to any particular RDBMS. Through the use of code-generation templates, multiple databases (Oracle, SQLServer, and the like), libraries (ADO, ODBC, and so on), and languages (such as C# and C++) can be the target. The data entities and data-access logic components are precompiled for higher performance, and the database schema can be validated against the application for greater type safety. The overall framework is simple and lightweight, and designed for ease of regeneration when the schema is updated. The tool itself is implemented in C#. It only runs on a Windows platform. It requires a suitable OLE DB driver that can provide the most comprehensive metadata. But this doesn’t mean the template and code are subjected to any of these conditions. Once the code generation is complete, any generic driver may be used by the application during runtime. The template can be in any programming language. It can use any data-access library, coding style, and software-design pattern. The tool may even be used to generate database scripts; for example, the scripts to create routine sequence and triggers. The bottom line is that, as long as you can do something once and know how to abstract the task into a template, the generator can fill in the rest. Code Generation of Classes for Single Tables Data entities that map to each single table are the most often used components in
26
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
a data-access tier. These components are normally implemented as classes in a procedural language, such as C++, Java, and C#. The class members, properties, and methods are often mapped to the columns of the corresponding table. These kinds of classes may or may not implement the DALC functionality as just mentioned. When using the DBAG tool to generate these kind of classes, you need to: 1. Let the tool connect with the database with the selected driver and read the schema information. 2. Provide your mapping preference, such as name mapping convention, type mapping convention, and class member access convention. 3. Provide your specific code template. After that, the generator produces the code you want. Figure 1 shows the main interface of DBAG. This interface shows the collected table information based on Oracle’s sample scott/tiger schema. The name, type mapping, and access control convention are stored in the DBAG.ini configuration file. Listing Three presents code template segments and the generated C# code fragments from this generator. DBAG is just a tool. It depends on you telling how to perform the task. The dictation is done through templates. DBAG provide a list of control commands and keywords to be used in your templates. These commands and key words are listed in Listing Four. A well-designed database requires that each table have an identifier (a natural key). In most cases, an identifier is also a key, but that’s not always the case. There are many tables that have an identifier but no key. Both the identifier and the key are important concepts in a database. Without them, it will be hard or inefficient for a database access tier to perform update and delete operations. Compared to an identifier, the key is a more widely recognized concept across relational databases. DBAG can get the key information from tables easily and pass it to the template. If you have con-
trol of the database schema, you may want to make sure that every table has at least the primary key defined. If a table does not have a natural key, you may want to create an artificial one. For example, create an Object ID (OID) column that gets its values from the database sequence, autonumber, or identity. In our experience, the artificial key is easier to use in a template because the names and types are more consistent. Data entity instances hold the data from/to table rows. Data-access logic components decide what rows are affected through the where clause of a SQL command. The data entity structures can be defined during code generation. During runtime, the dynamic where clause can be passed to the data-access logic as an argument. Generation of Classes for Table Joins Data entities that map to some joins are also often used in the data-access tier. Depending on your business process, sometimes these entities are preferred over the data entities that map to single tables. To generate classes that represent this kind of entity, you need to define the SQL statement to specify the join. This statement includes a selection part that gives each selected column a unique name and a condition part that specifies the joint criteria. The selection part is crucial because it determines the structure of the generated code. The condition part is not as important. As long as it can compose a syntactically correct SQL statement, it’s okay. At runtime you may provide the real joint relationship and parameters. The DBAG tool accepts the joint definition statements through the Sql2Class.xml file. On the GUI, DBAG lists the join-toclass maps together with the single-tableto-class maps. The only difference on the join-to-class map is that these maps have no table name values. You may need to consider this in your template. Generation of Classes for Stored Procedures A stored procedure may take none, one, or more parameters. It may return some
Listing One (a) For a single table
long _Deptno; string _Dname; string _Loc;
DEPT ( OID DEPTNO DNAME LOC )
bool bool bool bool
NOT NULL NUMBER(38), NOT NULL NUMBER(2), VARCHAR2(14), VARCHAR2(13)
(b) Corresponding class generated in C# public class Dept : DataAccessBase, IDept, IDataAccessBase { ... long _Oid;
http://www.ddj.com
Figure 1: DBAG schema mapping. selections, a single value, or nothing. This information can also be collected and made available for generating data entity wrappers. DBAG does not handle the stored procedure wrapper yet because it requires a totally different display interface and quite a different keyword set. However, based on the provided code, it should be easy to add such support. Sample Application The example application we provide with this article (available electronically; see “Resource Center,” page 4) implements a data-access layer for the famous Oracle sample database scott/tiger. To demonstrate some of the recommended concepts, we modified it to add an artificial primary key column OID to each table. We also rebuilt the relationships between the tables using the newly added columns. When generating the code, we used Microsoft’s OLE DB driver for Oracle and a template that combines database entity and database access logic. Besides demonstrating the normal database operations, it also demonstrates how to handle joints and how to bind SQL statements to the database schema. Conclusion Database access tier is an important component in enterprise applications. It helps make the business process code simple and independent. Although repetitive, it requires a considerable amount of time and attention from developers who otherwise can spend more time on their business logic. Computers are excellent at handling repetitive tasks. They are more powerful in repetitive tasks than humans, and it is wise to use them for generative programming tasks. DDJ
_OidNull = true; _DeptnoNull = true; _DnameNull = true; _LocNull = true;
public long Oid { get {return _Oid;} set {_OidNull = false; _Oid = value;} } ... }
(continued on page 28) Dr. Dobb’s Journal, December 2005
27
(continued from page 27)
_$MEMBER_NAME$ = fSql.GetNextColumnBytes(1024*200); //!!END IF!!// //!!END FOREACH MEMBER!!//
(c) For table joins SELECT DEPT.DNAME AS DEPARTMENT, EMP.EMPNO AS EMPNO, EMP.ENAME AS EMPNAME, EMP.SAL AS SALARY FROM DEPT, EMP WHERE DEPT.OID=EMP.DEPTOID ORDER BY DEPARTMENT
(f) Generated C# code
(d) Corresponding class generated in C#
_EnameNull = fSql.IsNextColumnNull(); _Ename = fSql.GetNextColumnString();
_OidNull = fSql.IsNextColumnNull(); _Oid = fSql.GetNextColumnInteger(); _EmpnoNull = fSql.IsNextColumnNull(); _Empno = fSql.GetNextColumnInteger();
public class DeptEmp : DataAccessBase, IDeptEmp { ... string _Department; long _Empno; string _Empname; long _Salary;
_JobNull = fSql.IsNextColumnNull(); _Job = fSql.GetNextColumnString(); _MgrNull = fSql.IsNextColumnNull(); _Mgr = fSql.GetNextColumnInteger();
public string Department { get {return _Department;} } ... }
_HiredateNull = fSql.IsNextColumnNull(); _Hiredate = fSql.GetNextColumnDateTime(); _SalNull = fSql.IsNextColumnNull(); _Sal = fSql.GetNextColumnInteger(); _CommNull = fSql.IsNextColumnNull(); _Comm = fSql.GetNextColumnInteger();
Listing Two (a) Raw SQL statement select emp.* from emp, dept where emp.deptoid=dept.oid and dept.dname='ACCOUNTING';
_DeptoidNull = fSql.IsNextColumnNull(); _Deptoid = fSql.GetNextColumnInteger();
(b) SQL built with generated information
Listing Four
... emp.Retrieve(Emp.DEPTOID == Dept.OID & Dept.DNAME == "ACCOUNTING"); ...
//!!FOREACH MAPPED TABLE!!// $CLASS_NAME$ $TABLE_NAME$ $GUID1$ $GUID2$ $MEMBER_COUNT$ $FOREIGN_KEY_COUNT$ //!!END FOREACH MAPPED TABLE!!// //!!FOREACH MAPPED JOINT!!// $CLASS_NAME$ $GUID1$ $GUID2$ $MEMBER_COUNT$ $SELECT_STRING$ $UPDATE_STRING$ $DELETE_STRING$ $INSERT_STRING$ //!!END FOREACH MAPPED JOINT!!//
Listing Three (a) Code Template columnNameStr = //!!FOREACH MEMBER!!// "$COLUMN_NAME$," + //!!END FOREACH MEMBER!!// //!!REPLACE Options=64; Count=1; Regex=/," \+/"/ !!// ;
(b) Generated C# code columnNameStr = "OID," + "EMPNO," + "ENAME," + "JOB," + "MGR," + "HIREDATE," + "SAL," + "COMM," + "DEPTOID" ;
(c) Code template //!!FOREACH MEMBER!!// bool Is$MEMBER_NAME$Null {get;set;} //!!END FOREACH MEMBER!!//
(d) Generated C# Code bool bool bool bool bool bool bool bool bool
IsOidNull {get;set;} IsEmpnoNull {get;set;} IsEnameNull {get;set;} IsJobNull {get;set;} IsMgrNull {get;set;} IsHiredateNull {get;set;} IsSalNull {get;set;} IsCommNull {get;set;} IsDeptoidNull {get;set;}
(e) Code template //!!FOREACH MEMBER!!// //!!IF MEMBER_TYPE=long!!// _$MEMBER_NAME$Null = fSql.IsNextColumnNull(); _$MEMBER_NAME$ = fSql.GetNextColumnInteger(); //!!END IF!!// //!!IF MEMBER_TYPE=double!!// _$MEMBER_NAME$Null = fSql.IsNextColumnNull(); _$MEMBER_NAME$ = fSql.GetNextColumnDouble(); //!!END IF!!// //!!IF MEMBER_TYPE=string!!// _$MEMBER_NAME$Null = fSql.IsNextColumnNull(); _$MEMBER_NAME$ = fSql.GetNextColumnString(); //!!END IF!!// //!!IF MEMBER_TYPE=DateTime!!// _$MEMBER_NAME$Null = fSql.IsNextColumnNull(); _$MEMBER_NAME$ = fSql.GetNextColumnDateTime(); //!!END IF!!// //!!IF MEMBER_TYPE=byte[]!!// _$MEMBER_NAME$Null = fSql.IsNextColumnNull();
28
//!!FOREACH MEMBER!!// //!!FOREACH PRIMARY_KEY!!// //!!FOREACH FOREIGN_KEY!!// //!!IF MEMBER_NAME=?!!// //!! ... !!// //!!IF COLUMN_NAME=?!!// //!! ... !!// //!!IF MEMBER_TYPE=?!!// //!! ... !!// //!!IF COLUMN_TYPE=?!!// //!! ... !!// //!!IF KEY_TYPE=?!!// //!! ... !!// //!!IF MEMBER_ACCESS=?!!// //!! ... !!// //!!END IF!!// $MEMBER_NAME$ $COLUMN_NAME$ $KEY_TYPE$ $MEMBER_TYPE$ $COLUMN_TYPE$ $MEMBER_ACCESS$ $COLUMN_ORDINAL$ $COLUMN_MAXLENGTH$ $COLUMN_TYPE$ //!!END FOREACH MEMBER!!// //!!END FOREACH PRIMARY_KEY!!// //!!END FOREACH FOREIGN_KEY!!// //!!FOREACH PARENT!!// $CLASS_NAME$ $TABLE_NAME$ $PARENT_TABLE$ $PARENT_CLASS$ $COLUMN_TO_PARENT$ $COLUMN_OF_PARENT$ $MEMBER_TO_PARENT$ $MEMBER_OF_PARENT$ //!!END FOREACH PARENT!!// //!!FOREACH CHILD!!// $CLASS_NAME$ $TABLE_NAME$ $CHILD_TABLE$ $CHILD_CLASS$ $COLUMN_TO_CHILD$ $COLUMN_OF_CHILD$ $MEMBER_TO_CHILD$ $MEMBER_OF_CHILD$ //!!END FOREACH CHILD!!// //!!REPLACE Options=?; Count=?; Regex=/???/???/ !!//
DDJ Dr. Dobb’s Journal, December 2005
http://www.ddj.com
Month-Text Ordering T
allows Date, Datetime, Time, Timestamp, and Year fields. Month-text ordering is also useful for filesystem sorting. For example, you could have a 2005 Marketing Plan folder that contains files such as January.doc, February.doc, and so on. These files might not be created in order and they may be modified out of order. As a result, sorting by the last-modified date/time would not necessarily arrange them properly. Other examples of month-related files include progress reports, sales reports, and newsletters. Some of these may be edited months after they are created. Folders with month names could be useful for storing photos and magazine articles.
David runs FileJockey Software and can be reached at http://www.FileJockeySoftware .com/ or
[email protected].
MonthOrder Class The first design goal for this class was to allow for short and long month names. In addition, I decided to include support for the common abbreviation “Sept.” Because the month names are not expected to change, they can be sorted alphabetically, stored in an array, and found using a binary search. However, because hash tables are available in Java and MFC, I decided to use one to store the month names. How many entries should be placed in this hash table? In a shareware file manager that I haven’t completed, I use 12 entries that are keyed on the first three letters of the month names. To check if a word corresponds to a short or long month name, this program copies the first three letters, converts them to uppercase, and tests whether this key is in the table. If there is a match, it then tests if this word matches the beginning part of the corresponding month name and that the length of this word is three for any month, four for September, or the full length of the month name. For the MonthOrder class, I decided to use 24 table entries. The additional entries are the short names, except for “MAY,” plus “SEPT.” As a result, the testing logic is simplified. A potential month name is converted to uppercase, an existing trailing dot is removed, and the key is checked in the hash table. (The previous routine does not handle trailing dots since they are removed by a tokenizer.) See Listing One. The compareTo function in Listing Two starts by requesting month numbers for a pair of strings. If both strings are month names and their month numbers differ, these numbers are subtracted. If they
A month field-type that sorts in month order DAVID WINCELBERG o have a fighting chance of remembering relatives’ birthdays and anniversaries, I put together a database table with the necessary information. But in designing the table I ran into a problem: Which field-types should I select? The database I use, Corel Paradox, provides three date-related field-types — Date, Time, and Timestamp. None of these seem appropriate since I do not want to sort rows based on the years when people were born or married. Instead, I included Month Name, Day, Year, and Month Number columns. After entering the data, I sorted on the Month Number, Day, Last Name, and First Name columns. This worked but has two drawbacks — I had to enter month numbers that correspond to month names, and the Month Number column was displayed on the printout. I later found that I could define a date display screen format that suppresses year numbers. (This format does not apply to printing, exporting, or publishing in HTML.) I then combined the Month Name, Day, and Month Number columns into a Date column. The resulting display looks cleaner but makes adding to the table more difficult. The reason is that the Date column includes — but doesn’t show— year information. To add birthdays or anniversaries in future years, I will need to display the dates in a format that shows year numbers and enter new rows with the year set to 2005. I present an alternative in this article. Databases should support a month fieldtype that will sort in month order. In searching the Web, I found that databases have varying support for date-related field-types. Of those that I looked at, MySQL has the most such field-types. It
30
Dr. Dobb’s Journal, December 2005
match, their lengths are subtracted. This is unnecessary but makes the order look nicer. If only one string is a month name, it is placed first. By grouping month names above other strings, there are no inconsistencies. For example, sorting “January,” “February,” and “Hat” would lead to an unpredictable arrangement without grouping. If neither string is a month name, they are ordered by using a routine that is based on my article “Alphanumeric Ordering” (DDJ, October 2000).
“The first design goal for this class was to allow for short and long month names” The Java implementation of this routine differs from the C implementation in that substrings consisting solely of digits must be created for Integer.parseInt to work. The C function atoi can handle strings that start with digits and are followed by letters; see Listing Three. Sample Program MonthOrderApplet (available electronically, see “Resource Center,” page 4) demonstrates month-text ordering. To use it, load MonthOrder.html into a browser. The class files must be in the same folder as this file. Then, you could either add strings individually or you could add name sets. For the latter, check either or both of the checkboxes and press the Add Name Set(s) button. After checking the Long Month Names box and pressing this button, you will see what is in Figure 1. Then press the Sort button. The month names will be arranged as in Figure 2. You can add short month names and resort the list. Because the compareTo function in Listing Two orders strings based on length when they represent the same month, short names appear before full names; see Figure 3. Pressing the Remove Item(s) button removes selected items. The corresponding http://www.ddj.com
File and Folder Names Several month names have meanings in other contexts. For example, “march” and “may” are common words. In addition, “April,” “May,” and “June” are women’s names. This creates an interpretation prob-
lem. One resolution is to surround text with braces to signal that it should be interpreted as a potential month. Then, when a filesystem comparison function sees a string such as “{May},” it knows to look in a hash table to find the index of the text inside the braces. The other problem with mixing month names and other strings was already addressed. Month names are grouped above other strings so that a comparison function does not return inconsistent results to a sorting algorithm.
Figure 1: Alphabetically ordered long month names.
Figure 2: Month-text ordered long month names.
(continued from page 30) routine does not use an array of indexes. Instead, it looks at each item and deletes it if it is selected. Afterwards, either the count variable or the current index is updated; see Listing Four.
Month Field-Type A month field-type could contain short and long month names and compare them using the routines in Listings One and Two. An alternate approach is to cache the month indexes along with case and length information. One possible encoding in a 32-bit integer starts with the left-most byte being zero. The remaining bytes would contain a case code, a length, and a month index. Months encoded in this manner would be compared after applying a 0xFF mask to extract the month index. The case code would range from zero to two to represent mixed, upper, and lowercase, respectively. If the first letter of an input month name is in lowercase, assume that this name is in lowercase. Otherwise, if the second letter is in uppercase, assume that this name is in uppercase. The remaining circumstance is to assume that this name is in mixed case, as in “January.” The lengths would range from three to nine, and the month indexes would range from one to twelve. A display routine would subtract one from an index, use this value to select a month name from a string array, extract the required number of characters, and adjust the case, if necessary. If 16-bit integers are available, then each part of a month-field code could be put into a nybble (4 bits). Then the mask would be 0xF. Conclusion A month field-type would be helpful in database tables when day or year information is not useful for sorting or display. In a filesystem context, month-text ordering supplements alphanumeric ordering in that it also arranges file and folder names in a way that respects the patterns that people see in strings. DDJ
Figure 3: Month-text ordered short and long month names. 32
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
Listing One
val2 = -1; } if ((test = val1 - val2) != 0) return test; n1 = e1; n2 = e2;
import java.util.Hashtable; public class MonthOrder implements SortTest { Hashtable map = new Hashtable(); public MonthOrder() { MonthInfo[] monthList = new MonthInfo[24]; monthList[0] = new MonthInfo ("JANUARY", 1); monthList[1] = new MonthInfo ("FEBRUARY", 2); monthList[2] = new MonthInfo ("MARCH", 3); monthList[3] = new MonthInfo ("APRIL", 4); monthList[4] = new MonthInfo ("MAY", 5); monthList[5] = new MonthInfo ("JUNE", 6); monthList[6] = new MonthInfo ("JULY", 7); monthList[7] = new MonthInfo ("AUGUST", 8); monthList[8] = new MonthInfo ("SEPTEMBER", 9); monthList[9] = new MonthInfo ("OCTOBER", 10); monthList[10] = new MonthInfo ("NOVEMBER", 11); monthList[11] = new MonthInfo ("DECEMBER", 12); monthList[12] = new MonthInfo ("JAN", 1); monthList[13] = new MonthInfo ("FEB", 2); monthList[14] = new MonthInfo ("MAR", 3); monthList[15] = new MonthInfo ("APR", 4); monthList[16] = new MonthInfo ("JUN", 6); // MAY is above monthList[17] = new MonthInfo ("JUL", 7); monthList[18] = new MonthInfo ("AUG", 8); monthList[19] = new MonthInfo ("SEP", 9); monthList[20] = new MonthInfo ("SEPT", 9); // alternative monthList[21] = new MonthInfo ("OCT", 10); monthList[22] = new MonthInfo ("NOV", 11); monthList[23] = new MonthInfo ("DEC", 12);
}
}
} } } // One or more strings has ended if (n1 >= size1 && n2 >= size2) return 0; else if (n1 >= size1) return -1; else return +1; // simpleANCompareTo
Listing Four void removeButton_ActionPerformed (java.awt.event.ActionEvent event) { try { int count = itemList.getItemCount();
for (int i = 0; i < monthList.length; i++) { map.put (monthList[i].monthName, monthList[i]); } // MonthOrder
private int findMonth (String monthName) { if (monthName == null) return -1; // Prepares test string String key = monthName.toUpperCase(); int dot = key.indexOf ('.'); // trims off trailing dot if (dot != -1) key = key.substring (0, dot);
}
} else { // caseless match test = Character.toLowerCase (name1.charAt (n1)) Character.toLowerCase (name2.charAt (n2)); if (test != 0) return test; else { n1++; n2++;
}
for (int i = 0; i < count; ) { if (itemList.isIndexSelected (i)) { itemList.delItem (i); count--; } else i++; } } catch (Exception e) { } // removeButton_ActionPerformed
MonthInfo mi = (MonthInfo) map.get (key); if (mi == null) return -1; else return mi.index; // findMonth
DDJ
... }
Listing Two public int compareTo (String name1, String name2) { int m1 = findMonth (name1); int m2 = findMonth (name2); if (m1 == -1 && m2 == -1) return simpleANCompareTo (name1, name2); else if (m1 == -1) // month names appear before other items return +1; else if (m2 == -1) return -1; else if (m1 == m2) // not needed, but makes order look nicer return name1.length() - name2.length(); else return m1 - m2; } // compareTo
Listing Three private int simpleANCompareTo (String name1, String name2) { int size1 = name1.length(), size2 = name2.length(); int n1 = 0, n2 = 0, e1, e2; int val1, val2, test = 0; while (n1 < size1 && n2 < size2) { if (Character.isDigit (name1.charAt (n1)) && Character.isDigit (name2.charAt (n2))) { // Finds ends of numbers so that parseInt will work for (e1 = n1 + 1; e1 < size1 && Character.isDigit (name1.charAt (e1)); ) e1++; for (e2 = n2 + 1; e2 < size2 && Character.isDigit (name2.charAt (e2)); ) e2++; try { val1 = Integer.parseInt (name1.substring (n1, e1)); } catch (NumberFormatException ex) { val1 = -1; } try { val2 = Integer.parseInt (name2.substring (n2, e2)); } catch (NumberFormatException ex) {
http://www.ddj.com
Dr. Dobb’s Journal, December 2005
33
PROGRAMMER’S TOOLCHEST
Object-Relational Mapping in Java with SimpleORM A lightweight, yet powerful, ORM implementation MARTIN SNYDER AND TED O’CONNOR
O
bject-relational mapping encompasses a wide range of techniques and libraries for handling bidirectional serialization of runtime objects in an object-oriented system within a relational database. As is often the case with Java, a number of options are available, many of them open source. In this article, we examine a mature alternative named SimpleORM (http://www.simpleorm.org/), which was created and is maintained by Anthony Berglas. We have adopted SimpleORM as our primary database integration library for a number of reasons: • SimpleORM acts as a direct conduit to the underlying JDBC data source. While there are internal caching mechanisms for performance reasons, there are no intermediate storage mechanisms visible to the user of the library. SimpleORM operations have the effect of acting directly on the underlying database. • SimpleORM lives up to its name in terms of providing well-defined functionality in a small and clear package. The simpleorm.jar weighs in at 112 KB and has no dependencies. The implementation easily supports the web site’s claims that it is a practical option for developers to step into the library code to diagnose and fix any issues they might have. • SimpleORM is easy to configure, requiring few lines of code and no XML. • SimpleORM doesn’t do much. This may sound like a strange thing to desire in an integration library, but in our case Martin and Ted are senior technologists at Wingspan Technology. They can be reached at
[email protected]. 34
we appreciated SimpleORM doing what was required in a direct manner and leaving the rest to us. • SimpleORM can be compiled under both Java and J# (.NET). We wouldn’t expect this feature to be of particular importance in most cases, but we were able to take advantage of it in our own development on one project. To show how to use SimpleORM, we created a sample “Settings Storage” component. The source code for this component is available electronically (see “Resource Center,” page 4) and is part of the (upcoming) 2.19 release of SimpleORM. Our settings component uses a representation similar to a filesystem or the Windows Registry. Logically, the only type of object in the system is a folder. A folder can contain subfolders and properties. Properties are simple name/value pairs where the value can be any serializable object in Java. Infinitely deep folder hierarchies are supported, subject to database constraints. Data Model Authority One of the most important elements of an Object-Relational Mapping tool is the location and specification of the data model to be used. In any Object-Relational Mapping (ORM) system, there will be at least two models — one or more table schemas in the database, and one or more class definitions in the OO environment. There can potentially be more, especially if the definition of the data models and mappings is external to both the programming language and database. The best case for any ORM is when the datamodel authority is used to generate everything else. While this is easily achieved for new applications, it is often impossible when integrating with an existing system or code base. In SimpleORM, the authority is built within the Java classes themselves. If desired, this mapping can be used to generate database tables or the mapping can be described in such a manner as to map to an already existing table. Users will immediately note how no effort is made to shield the user from database concepts. Dr. Dobb’s Journal, December 2005
This, in part, illustrates the intention of SimpleORM to act as a direct conduit between the caller and the database. Listing One shows two simple SimpleORM objects. SimpleORM objects inherit from SRecordInstance and include an SRecordMeta object. SRecordMeta objects are the heart of the SimpleORM system and contain the mappings used at runtime to present rows in database tables or queries as Java. By convention, the SRecordMeta is typically defined as a static member of the Java class it represents.
“SimpleORM acts as a direct conduit to the underlying JDBC data source” This is not a strict requirement, but it does enable certain conveniences and there is no compelling reason to do otherwise. Next we define the fields of the object (and columns of the corresponding table). Note that each field constructor takes the SRecordMeta as its first argument. The fields automatically register themselves with the SRecordMeta in their constructors, and are registered in the order they are defined in the source file. Order is generally not important, though all primary key columns must appear before all others. At least one primary key field is required for all objects as SimpleORM requires that all objects be uniquely identifiable by the system. SRecordInstance is an abstract class with one abstract method getMeta( ). Given an SRecordMeta, it is trivial to determine the Java class associated with it. Using this method, it is trivial to perform the reverse operation — to determine which SRecordMeta is bound to an arbitrary SimpleORM object instance. No reflection is required. Further examination of Listing One illustrates a handful of SimpleORM mechanisms. Each data type in the system is http://www.ddj.com
represented by a different SField* class. All major data types are supported by SimpleORM, and it is trivial to add additional user-defined SField definitions, though out of the scope of this article. Various flags are available to modify the field mappings. The SFD_GENERATED_KEY mapping indicates that the database should be used to generated values for the specified field. This only works for Integer columns and its implementation is database dependent. SimpleORM does not currently support optimal implementations for many databases (PostgreSQL is one for which it supports this mechanism very well), but this facility is the target of significant work in the upcoming release. The SFD_MANDATORY flag is used to indicate that NULL values should not be accepted for this field. There are of course many other options available. The SFieldReference field mapping deserves special mention, as it is used to represent foreign key references in the database. SimpleORM supports an elegant mechanism for traversing foreign key references. Connection Management & Transactions Because of the various environments where SimpleORM might be used, creation of connections is deferred to the caller. This means that the caller is free to use any internal or external connection allocation or pooling mechanism. Such a mechanism might be provided as part of an application framework or J2EE server or can be trivially created. Listing Two demonstrates the steps involved in initializing and terminating SimpleORM using the vanilla JDBC to create a database connection. Static methods on the SConnection class are used to bind the connection to the SimpleORM library, control a transaction, and release the connection. Internally, SimpleORM binds the JDBC connection to the current thread, meaning that all of the code elements in Listing Two must be called by the same thread. Any SimpleORM invocations made by that thread in the interior of the requested transaction is automatically routed to the appropriate connection. Though it may not appear so at first glance, this mechanism is thread safe due to SimpleORM’s internal usage of Thread Local Storage mechanisms. In fact, because of those mechanisms, SimpleORM is able to defer almost all locking mechanisms to the database itself. Simultaneous threads executing within SimpleORM each have their own connections and transactions; they each execute within their own operating contexts. http://www.ddj.com
Object Management Database-mapped objects in SimpleORM are all handled through the SRecordInstance object interface. This interface can be extended via subclassing if desired, but it is entirely possible to utilize SimpleORM with nothing more than Listing One. Object creation and loading is handled via a single method on SRecordMeta, findOrCreate( ). FindOrCreate takes as its parameter a single object or an object array representing the primary key value(s) of the object being requested. If the primary key is designated to be generated by the database, then new objects can be created with the createWithGeneratedKey( ) interface. Listing Three demonstrates some common interactions we might perform on our example Folder object. We begin by loading a folder with an ID value of 5 and creating a second folder with a newly generated ID. We can either cast these objects to be our defined type Folder, or deal with them as generic SRecordInstance objects. We then go on to request the name of folder1 and the parent of folder2. Note how we use specific, type-based get routines of the SRecordInstance class. These routines provide additional data validation both in ensuring that the SField instance is of the appropriate type and is valid in the current transaction and also that the value retrieved from the database adheres to the declared expectation. In the case of a foreign-key reference, the referenced object is automatically loaded and returned, or retrieved from the transaction cache if possible. For most primitive data types, we can retrieve the data any number of ways and SimpleORM will coerce the actual value into the requested type. We do this with the ID field, requesting it as three different data types. It is also trivial to define and use methods such as getName( ) if that is preferred as a matter of style, but there is no need to do so. When data is requested from the database (such as during the object load), the attached connection is immediately used to retrieve the appropriate values. This data is then loaded into the SRecordInstance-derived object. Unless specifically directed otherwise, SimpleORM never reloads a data value during a single transaction, so repeated requests for the same data return the value cached in the SRecordInstance and do not touch the connection a second time. For this reason, SRecordInstance objects are only valid within the context of the SimpleORM transaction that created them. This aspect initially caused us great discomfort and can, for some developers, force you to rethink how your application should interact with the database. It is the nature of many developers to think of the database Dr. Dobb’s Journal, December 2005
as a storage mechanism like a filesystem. Data objects are saved and loaded from the database and an ORM simply facilitates that process. SimpleORM takes a different approach, where you are either talking to the database or you are not. The data lives only in the database, and you use SimpleORM as a conduit to facilitate that communication. During our own education, we found that understanding both models leads to a much deeper understanding of ORM technology in general. SimpleORM is a great choice if you want your application to directly manipulate your database. There are parallel set routines for all of the get routines with the same type signatures. Set operations manipulate the object cache immediately, and the database is updated en-masse when a transaction is committed or when SConnection.flush( ) is called. A unique object in the database is only ever loaded into the object cache for a transaction once. This means that if you load an object, navigate to it via foreignkey references, and retrieve it from a query’s result set, the exact same object instance will be returned each time. Object deletion is accomplished by calling the deleteRecord method on any SRecordInstance object. This method can be overridden as in Listing Four so long as the base implementation is invoked. This would typically be done to prevent any foreign-key violations that might result from deleting an object. It is always faster to delete or update several related records at once using a single SQL Delete statement than to fetch them into memory using JDBC and delete them one by one. SimpleORM supports this by enabling records for a specific table to be purged from memory, and so maintaining a consistent view of the data. (Most databases also support Cascade Delete options, but they can be very confusing for both Java and stored procedure code.) Queries Listing Five illustrates usage of the SimpleORM query API to iterate over all folder objects at the root of our hierarchy. The newQuery method on SRecordMeta creates a new query object designed to return objects of the type represented by SRecordMeta. There are a great number of operators available, including the isNull method we use here. There are also numerous type-specific gt (greater than), lt (less than), and eq (equals) methods. Conjunction operations (and and or) are available as well. The eq method can directly support references to other tables, and more general table joins are also supported, but our example is not well suited to demonstrate this advanced functionality. Listing Six demonstrates another usage of the query API. In this case, our queries 35
are designed to return exactly 0 or 1 records. Note how isNull, and, and eq are chained together mimicking the equivalent SQL. The SimpleORM query API is an extremely thin layer on top of SQL. This mechanism lends itself to rapid adoption by developers experienced with SQL queries. It is also possible to manually specify the SQL fragment that forms part of the where clause, which is occasionally useful for less common or database specific queries. A further mechanism enables the entire SQL query to be specified with the cache being flushed manually if needed. Rules Enforcement SimpleORM supports rules enforcement via a series of overridable methods on the SRecordInstance class. By far, the most common use of this model is in the area of field validation. validateField is called each time a field value is set. validateRecord is called once immediately before an object is written to the database. A violation is signaled by throwing an SValidationException.
In practice, it is better to use validateField wherever possible, as this will cause an exception to be thrown at the exact location where the offense is committed. validateRecord is called only when SConnection.flush( ) is called, or when a transaction is committed. As a result, it may not be immediately obvious when the infraction occurred. Of course, in many cases, per-field validation is not sufficient. For example, the possible values for one field might be dependent on the setting of another field. To avoid chicken-andegg type problems, it is better to simply validate the entire record at once rather than try to do it piecemeal as the fields are being set. Listing Seven shows a simple validation routine applied to the Folder’s name field. Here we see one of the benefits of having our ORM metadata (SRecordMeta and SField* entries) defined as static members of the object implementation (SRecordInstance). With a simple reference equality check we can quickly determine which field is being modified and apply whatever logic we wish. Because we defined this field with the SFD_MANDATORY flag,
Listing One
static SFieldInteger ID = new SFieldInteger(s_meta,"ID",new SPropertyValue[] { SSimpleORMProperties.SFD_PRIMARY_KEY, SSimpleORMProperties.SFD_GENERATED_KEY }); static SFieldReference PARENT = new SFieldReference(s_meta,s_meta,"PARENT"); static SFieldString NAME = new SFieldString(s_meta, "NAME", 64, SSimpleORMProperties.SFD_MANDATORY); public SRecordMeta getMeta() { return s_meta; } } public static class Property extends SrecordInstance { static SRecordMeta s_meta = new SRecordMeta(Property.class, "PROPERTY"); static SFieldReference PARENT = new SFieldReference(s_meta, Folder.s_meta, "PARENT", new SPropertyValue[] { SSimpleORMProperties.SFD_PRIMARY_KEY, SSimpleORMProperties.SFD_MANDATORY }); static SFieldString NAME = new SFieldString(s_meta, "NAME", 64, new SpropertyValue[] { SSimpleORMProperties.SFD_PRIMARY_KEY, SSimpleORMProperties.SFD_MANDATORY }); static SFieldObject VALUE = new SFieldObject(s_meta, "VALUE"); public SRecordMeta getMeta() { return s_meta; } }
Listing Two Connection conn = DriverManager.getConnection("jdbcURL", "user", "pass"); SConnection.attach(conn, "Connection Name"); SConnection.begin(); ... SConnection.commit(); SConnection.detachAndClose();
Listing Three Folder folder1 = (Folder)Folder.s_meta.findOrCreate(new Integer(5)); Folder1.assertNotNewRow(); SRecordInstance folder2 = Folder.s_meta.createWithGeneratedKey(); String name = folder1.getString(Folder.NAME); Folder parent = (Folder)folder2.getReference(Folder.PARENT);
folder1.deleteRecord();
Listing Four public static class Folder extends SRecordInstance { ... public void deleteRecord()
36
Conclusion SimpleORM is a lightweight, yet powerful ORM implementation available under a liberal open-source license. SimpleORM makes little, if any, attempt to shield the user from database concepts, so developers with existing database knowledge and experience will have more success when using the library. The code base is similarly lightweight and easy to extend, though for most purposes, this will not be necessary. DDJ
{ // Delete all subfolders SResultSet subFolders = s_meta.newQuery().eq(PARENT, this).execute(); while (subFolders.hasNext()) subFolders.getRecord().deleteRecord();
public static class Folder extends SRecordInstance { static SRecordMeta s_meta = new SRecordMeta(Folder.class, "FOLDER");
Object idObject = folder2.getObject(Folder.ID); String idString = folder2.getString(Folder.ID); Int idInt = folder2.getInt(Folder.ID);
null values will not be permitted. That enforcement is only done when the object is written to the database at the end of a transaction, so we can add an additional check here to provide for a more timely error. In our full implementation of the example, we wish to be able to represent a folder hierarchy with a path string. Blank folder names and folder names containing our path delimiter character make parsing a path string more difficult, so we can simply disallow them here.
// Delete all properties of this folder SResultSet props = Property.s_meta.newQuery().eq(Property.PARENT, this).execute(); while (props .hasNext()) props.getRecord().deleteRecord(); // Invoke the base implementation super.deleteRecord(); }
Listing Five SQuery query = Folder.s_meta.newQuery().isNull(Folder.PARENT). ascending(Folder.NAME); SResultSet rs = query.execute(); while (rs.hasNext) { Folder folder = (Folder)rs.getRecord(); ... }
Listing Six SQuery query = Folder.s_meta.newQuery() .isNull(Folder.PARENT) .and() .eq(Folder.NAME, "Test"); SResultSet rs = query.execute(); Folder folder = (Folder)rs.getOnlyRecord();
Listing Seven public static class Folder extends SRecordInstance { ... public static final String DELIMETER = "/"; public void validateField(SFieldMeta field, Object newValue) { if (NAME == field) { if (null == newValue || "".equals(newValue) || (newValue.toString().indexOf(DELIMETER) >= 0)) { throw new SValidationException("Folder name cannot be blank or contain " + "the DELIMETER character: " + DELIMETER); } } } }
DDJ Dr. Dobb’s Journal, December 2005
http://www.ddj.com
Building Grid-Enabled Data-Mining Applications Tackling big data-mining jobs with parallelization ALEX DEPOUTOVITCH AND ALEX WAINSTEIN
D
ata mining is the process of extracting information from large volumes of data. By its very nature, it is a computationally intensive exercise. Because data volumes are doubling annually (according to a Winter Corporation survey, http://www.wintercorp.com/WhitePapers/ TTP2003_WP.pdf), it can be difficult to achieve even small performance improvements simply by tweaking known datamining algorithms. Of course, sampling the input data can help, but you pay the price of reduced accuracy, which is unacceptable for many tasks. Increasing the power of hardware doesn’t offer much help; CPU clock frequencies and hard-drive datatransfer rates both have fixed upper boundaries, and no matter how much money you spend, you cannot overcome them. While you cannot improve the speed of a single CPU, you can increase their
Alex Depoutovitch is the technical team lead and Alex Wainstein is director of technology and production at Generation5 Mathematical Technology. They can be contacted at
[email protected] and
[email protected], respectively. http://www.ddj.com
number. This is why multithreaded processor cores, multicore chips, and multiprocessor servers are becoming common. Although this leads to better performance, the total number of CPUs on a single server is still limited. Although today’s dual-CPU server price is reasonable, the cost of the 32-CPU servers can exceed the price of a private jet airplane. Still, there are network- based alternatives which hold promise for high- end data mining, foremost among them “computing grids” that enable cooperative processing of single or multiple tasks by different computers. The term “computing grid” designates a group of connected computers that combine resources to work together, appearing to users as a single virtual computer. The main difference between “grids” and “clusters” is that computing clusters usually contain only homogenous computers, whereas computers in a grid may have totally different hardware and software configurations. There are two ways you can use computing grids: • To evenly balance incoming requests among individual computers so that each request is served by one computer. • To speed up the processing time of incoming requests by executing each of them in parallel on several computers in a grid. Computing grids have centralized or decentralized scheduling. With centralized scheduling, a dedicated computer distributes requests and collects results. Decentralized scheduling, on the other hand, involves a cooperative process Dr. Dobb’s Journal, December 2005
among computers in a grid. Decentralized scheduling provides better availability and fault tolerance, but produces more network traffic, is less flexible, and more difficult to implement.
“Designing parallel algorithms is a complex procedure”
G5 MWM Grid from Generation5 is a data analytics application that lets you build and manage computing grids specifically for data-mining tasks. G5 MWM Grid achieves considerable performance improvements by executing various datamining tasks on a computing grid in multiuser environment. G5 MWM Grid, which uses centralized scheduling, consists of the components in Figure 1. The Central node is responsible for the scheduling and coordination of processes running on the Calculation nodes. There is only one Central node in a grid. Users prepare “jobs” and communicate with the Central node through grid client software (Figure 2). The client software lets users monitor and control job execution progress and 41
examine and change grid configurations. User requests are sent to the Central node, which divides them into chunks that can be executed in parallel, sending these chunks for execution to all available Calculation nodes. Calculation nodes are responsible for the execution of tasks received from Central node. They report back to the Central node about execution progress and completion. Exchanging large chunks of data among the Central node, Calculation nodes, and client software requires shared data storage, such as a database server or shared filesystem. Parallelization of Algorithms In general, designing parallel algorithms is complex and not always possible. That said, data mining is one area that often lends itself to parallelization. The most important measure of how well a given algorithm can be parallelized is “scalability” or the dependency between performance of a grid and the number of nodes in it. While the ideal case is a straight line, in real life there is a threshold value after which adding more nodes to the grid does not improve performance (Figure 3).
This threshold value differs greatly, depending on the specific algorithm. When designing parallel algorithms, you often encounter situations where part of the calculations can be done only sequentially, and to proceed further, you need the results of the calculation. In this case, you have two options: execute this critical part on one node and make other nodes wait for its completion, or redo this part on each node. If the nonparallelizable part is small, it is a good idea to repeat it on each node to achieve better scalability. Another question that comes up when designing parallel algorithms is how to provide effective concurrent access to shared data. If any node may update any part of the data at any time, data access must be synchronized and the scalability of algorithms is significantly affected. Even if all nodes only read data, the datareading device becomes a bottleneck sooner or later. However, there are techniques that address this problem. If you do not update the data and the time required to copy the data from the shared storage is not significant, you can create a temporary local copy of the shared data
42
• Avoid parallel chunks that depend upon the results of other chunks. • Shared data access must be minimized (especially write access). • Message passing between the Central node and Calculation nodes must be minimized. Parallelization of Data-Mining Algorithms In data mining, “prediction” is the task of producing unknown data values based on some previously observed data. Input data for prediction is divided into two parts — one containing training data and the second containing target data that you need to predict. Both parts have the same columns (variables). Variables are divided into two groups (Figure 4): independent and dependent. Independent variable values are provided in both training and target parts, while dependent variables are known only in the training part. The goal is to predict dependent variables in target data. There are many prediction methods available, including: • Decision trees in which you process training data to receive a set of rules such as: “if age more than 30 and income more than 90K and number of children less than 2, then most probable car of choice would be Mercedes”. Using these rules data is predicted.
Figure 1: G5 MWM Grid architecture.
Figure 2: Users prepare “jobs” and communicate with the Central node through grid client software.
on each node on which chunks are processed. In this case, data access speed is no longer a limiting factor for scalability. Even when some data must be updated by several nodes, it often is possible to do it without synchronizing access. For example, consider a situation when you need to calculate the average value of some array using a grid of n calculation nodes. You can divide this array into n equal chunks. Each node will be given its chunk and calculate the sum of values in this chunk and report back to the Central node. After all nodes are finished, the Central node sums up all received values and divides by the array size. This way, you avoid any data that must be shared between nodes. With this in mind, requirements for parallel algorithms include:
Figure 3: Scalability. Dr. Dobb’s Journal, December 2005
Figure 4: Data for prediction. http://www.ddj.com
• Linear regression, which is based on training data you build and linear formulas to calculate dependent variables in target data using independent variables. • Nearest neighbor method. This method, by maintaining a dataset of known records, finds records (neighbors) similar to a target record and uses the neighbors for classification and prediction. This method is memory-based and requires no model to be fit. The first two methods require lots of preparation time and cannot be parallelized without making big changes in their algorithms. However, the third method is attractive because you can do minimal preprocessing and predict each record independent of others, so no modification is needed. With real-world problems, the number of records varies from thousands to hundred of millions. This method is particularly well suited for scalability. The algorithm is described like this:
any methods that can effectively parallelize feature selection for a single dependent variable. But often it is the case that you have several dependent variables. In this case, each dependent variable can be processed on separate Calculation nodes, in parallel. Validation lets you estimate how good the prediction is. To do this, you only need training data. You pretend that you do not know some part of it and try to predict it based on the rest of the records. Then you estimate the error of prediction. This is usually tried several times, predicting different parts of training data. Although it is possible to parallelize Validation the same way as with Prediction, we found that with typical parameters — several thousands of rows and 5 – 20
different tries — it is more effective to execute different tries on different nodes. Here, depending on the number of tries, we achieve scalability up to 5 –20 Calculation nodes. Implementation The main goal of the MWM Grid design was to build a simple, but flexible,
Figure 6: Scheduler.
1. Divide the target data into the desired number of nonoverlapping chunks. The number of these chunks must be great enough to take advantage of all the computational nodes available. On the other side, the chunks should be big enough so that the scheduling and preprocessing overheads are not significant compared to single data chunk processing time. 2. Copy training data to each Calculation node. 3. Process training data on each node. 4. Assign a chunk of target data to each Calculation node and perform predictions against the assigned chunk. Feature selection algorithms are used to determine the minimal subset of variables upon which the variables of interest depend. Currently, we do not know
Figure 5: Workflow in the Central node. (1) job request from client is authenticated; (2) request is submitted to Scheduler; (3), (4) Scheduler asks Job Splitter for next chunk; (5) chunks are sent to Calculation node; (6) upon chunk completion, chunk results are submitted to Results Summarizer; (7) Results Summarizer checks with Scheduler, and if all chunks are finished, it submits results back to client. http://www.ddj.com
Dr. Dobb’s Journal, December 2005
Dobbs_0508.indd 1
43
07.06.2005 17:24:19 Uhr
framework for grid computations that support a variety of data-mining algorithms. The framework needed to be easily extensible for adding new algorithms without changing core functionality, while reusing as much of the existing libraries and OS mechanisms as possible. We implemented G5 MWM Grid on Windows using Microsoft .NET for the front end and C++ for the mathematical algorithms and Central and Calculation nodes. The UI consists of: the Project Editor, a wizard-like application for creating data-mining projects to be executed as jobs; the Monitor, for remote monitoring and managing job execution; and the Grid Administration, an application for grid management and configuration. Both the Central node and Calculation nodes operate as Windows services. During startup of the Central node, the program reads the cluster configuration from the disk, and starts and connects the Calculation nodes. All communications are done through a SOAP-compliant protocol over named pipes for easy integration with other products. From the developer’s perspective, the Central node consists of five different parts — a User Authentication Module, Job Splitter, Task Scheduler, Communication Library, and Results Summarizer. Because we’ve integrated our security model with Windows, the system administrator defines an access control list (ACL) of groups of users with associated permissions — execution/termination of jobs, adding/removal of Calculation nodes from the grid, and so on. The system will identify the user who attempts to connect to the Central node, identify the groups it belongs to and determine privileges. Windows-integrated security eliminates the development issues of securely storing passwords; it also eliminates the sysadmin hassle of using yet another set of tools and maintaining yet another list of users; and the users are happy they don’t need to remember and enter different passwords for their Windows and grid accounts. The Job Splitter module is responsible for splitting jobs into chunks. Each type of job — prediction, feature selection, and so on — has its own implementation. The general principles, according to which it works, were described earlier.
One of the most important parts of the grid is the Task Scheduler. Its purpose is to: • Simultaneously handle jobs with different priorities. • Support dynamic change of a job priority. • Allocate specific resources for any job. • Support heterogeneous grid configurations (when the nodes have different processing power). • Support dynamic changes in the grid configurations (add/remove nodes) • Limit the number of active nodes, reserving the rest for other activities, if required. The first three items give us flexible job management, such as bumping up priority of an urgent job or ensuring that some job will finish by a fixed deadline. The rest gives the opportunity to use the idle time of the existing computers for the grid purposes and share grid computer resources with other applications. The elementary resource that the Scheduler operates on is the Processing Unit (PU), which represents a virtual processor on the Calculation node. Each node may have one or more PUs. The number of PUs for each node can be bigger or smaller than the physical number of CPUs. For example, if in addition to the grid computations, you want to be able to execute some other programs on a multiCPU Calculation node, you specify a PU number that is less than the physical number of processors. When you have processors with hyperthreading or you are going to execute I/O-intensive jobs, then you can specify more PUs than physical CPUs available. The elementary task that the Scheduler assigns to resources is a job chunk. The Scheduler has six different priorities for jobs. For each priority, there is a separate queue of jobs. When a job is assigned priority, it is placed into a corresponding queue. As soon as there is a free PU, the scheduler checks job queues starting with queue with highest priority. A chunk from the first job found is sent for execution to the free PU and the job is placed to the end of the queue (Figure 6).
Model 1
Model 2
Model 3
Training set size (rows)
40,000
250,000
10,000
Target set size
40,000
250,000
15,000,000
Independent variables number
22
47
49
Dependent variables number
1
5
1
Total number of missing values
40,000
1,250,000
15,000,000
Table 1: Model parameters. 44
Dr. Dobb’s Journal, December 2005
In addition to priority, a job may have the maximum number of PUs available for a particular job, which means that the job will have no more than the specified number of chunks executing in parallel. In this case, if you want some job to finish by a fixed time, you make its priority higher than other jobs and limit the number of PUs available for it. This way, a fixed set of resources is secured for the job. All communication between the client software and the Central node, as well as between the Central node and Calculation nodes, are done using SOAP. The underlying transport protocol is the Windows implementation of named pipes. It is a message-oriented protocol with guaranteed delivery and integrated authentication. Our original implementation was one named pipe using the “pull” approach. Thus, when the client wants to refresh its information, it queries the server about its current state (from the point of view of the Calculation node, the Central node is the client because it sends commands). But this approach turned out to be inefficient and not too scalable as the number of Calculation nodes increases: The delays with response were dozens of seconds when the number of nodes exceeded 10. Whenever the user needs to see a complete picture of the cluster state, it issues a request to the Central node; and the Central node sends requests to all Calculation nodes, waits for answers, and then sends a summary back to client. Because of this, we implemented a new “push” approach so that whenever the state of a Calculation or Central node changes, it fires off a notification for interested parties. This proved to be effective and produced no noticeable delays, but required the use of two pipes — one for commands and responses, and one for asynchronous events. These events include completion of a chunk or job, cluster configuration change, and so on. This architecture resembles DCOM, but it has the advantages of being an open, high-level protocol that is easily extensible and does not require complex configuration. For convenience, the SOAP protocol implementation can be hidden behind COM or C++ classes. Results Summarizer The Results Summarizer implementation depends on the data-mining task. In any case, it is crucial to provide shared storage (such as a database server or shared disk). For example, with prediction, the Summarizer uses a database server hosting the target data to update it from different Calculation nodes. With feature selection, the results for each chunk are http://www.ddj.com
stored in the filesystem at a shared location, and after all chunks are done, the results are consolidated into the final report. Because it is not known in advance on which node the job will execute, each should have access to the shared storage. Because there’s always a risk that one computer in the grid will fail, the G5 MWM Grid provides a way to recover from this situation. In case of a node failure, it removes the defective node from the grid and transfers the task to another node to redo, whenever possible. The new nodes can be dynamically added to and removed from the grid. Adding is straightforward: After registration through the UI, it immediately becomes available to the Scheduler. The removal is more complex. If the node is idling, it can be removed immediately upon request. However, when some of its PUs are occupied by the task, no more tasks will be assigned to it and it will be removed upon completion of all tasks currently being executed. Performance To evaluate performance and scalability of G5 MWM Grid, we executed several prediction projects on a grid with a number of Calculation nodes varying from 1 to 10. We used two different sets of data (Model 1 and Model 2) that differ by size of training and target datasets, and the number of variables. All parameters are listed in Table 1. The speed of execution is measured in predicted values per minute. We calculated it with a different number of Calculation nodes in the grid. Computers that we used were singleCPU 1-GHz Pentium III's workstations running Windows 2000 Professional. The Central node and the database shared one Pentium 4 1.8 GHz. The computers were on 1-GB network. The results are summarized in Figure 7 and demonstrate the nearly linear scalability. It does not show any sign of saturation. Our measurements showed that the average CPU load on the Central node was around 30 percent and the network utilization under 5 percent, meaning the grid size can still be increased several times, even with hardware that can be found in any office. Model 3 (see Figure 7) was used for the high-end benchmark. The grid was implemented with six Calculation nodes, each equipped with dual Intel Xeon 3.0GHz CPU on the 1-GB network. The database server was a 2.4-GHz quad-CPU server. In this configuration the grid was able to predict 15 million values in 5 hours, at the rate of around 50,000 predicted values per minute. Without grid technology, a similar performance can be achieved by using a million dollar’s worth of mainhttp://www.ddj.com
Figure 7: Scalability testing. frames. The G5 MWM Grid did that on less than $100,000 worth of hardware. Conclusion Computing grids bring many advantages to data mining. This is because data-mining applications are particularly resource hungry; the amount of data available (and so the computational power required to process it) continues to grow exponentially.
Dr. Dobb’s Journal, December 2005
In an attempt to improve accuracy, the algorithms become more complex and require even more computational power. And grid computing provides the computational power required. Because many data-mining tasks can be effectively parallelized, the grid is their natural execution environment. DDJ
45
The OCAP Digital Video Recorder Specification Building DVR applications LINDEN DECARMO
T
he instant a DVR is released, hordes of hackers descend on it to figure out how to add additional hard-drive capacity and tweak the software. Hacking DVRs is popular because manufacturers don’t use open APIs and discourage software tweaking of their DVRs. Fortunately, CableLabs (the R&D consortium for the cable television industry) has published the OCAP Digital Video Recorder (DVR) specification that defines an open API that should minimize the need to hack future DVRs based on this technology. In this article, I examine the OCAP DVR spec and show how you use it to create DVR applications. Monopoly Versus Competition The software in virtually all cable DVR platforms is proprietary and highly secretive (the exception being TiVo’s HME; see “Building on TiVo”, DDJ, March 2005). To Linden is a consultant engineer at Pace Micro Technologies Americas where he works on DVR software architecture. He is the author of Core Java Media Framework (Prentice-Hall, 1999). He can be contacted at
[email protected]. http://www.ddj.com
create applications for these proprietary platforms, you usually must sign a NonDisclosure Agreement (NDA) and pay DVR vendors tens of thousands of dollars before obtaining an SDK. Hackers typically don’t have thousands of dollars to spend and often won’t sign NDAs as a matter of principle. Consequently, they resort to other means of tweaking the software. At the same time, the cable industry is trying to transition from legacy cable boxes with proprietary APIs to a Java-based API set called the “OpenCable Application Platform” (OCAP); see my article “The OpenCable Application Platform” (DDJ, June 2004). While the initial OCAP specification has a rich API set that can control High Definition (HD) and advanced captioning features found in modern settop boxes, it is sorely lacking Video Recording (VR) functionality. Fortunately, this was not an intentional omission. The OCAP spec was released before DVRs became mainstream devices and CableLabs did not try to cram a preliminary DVR specification into the standard before it was solidified. In the years since the original OCAP spec was published, HD DVRs have exploded onto the market and became key revenue generators for many cable vendors. Consequently, CableLabs realized that the addition of a Java-based DVR API was an essential if OCAP wanted widespread acceptance. Consequently, it published the OCAP Digital Video Recorder (DVR) specification (http://opencable .com/downloads/specs/OC-SP-OCAPDVR-I02-050524.pdf). Dr. Dobb’s Journal, December 2005
Bare Minimum CableLabs has three goals for OpenCable: • Define the feature set of next-generation hardware. • Foster competition based on open standards. • Enable boxes to be sold at retail locations.
“The first thing a DVR application typically does is detect the storage devices connected to the platform” To further these goals, CableLabs created OCAP— a middleware API that is operating system, hardware, and network neutral. By eliminating all proprietary operating systems and conditional access technologies, OCAP ensures competition by defining an open standard and allowing vendors to innovate based on their particular expertise. CableLabs has adopted a similar approach with the OCAP DVR extension. Rather than taking an authoritarian approach and forcing all vendors to create 49
clone DVR products, the DVR spec defines the minimum set of standards that any OCAP DVR platform must support. Vendors can differentiate themselves via enhanced functionality, price, or time to market. CableLabs only insists that each OCAP DVR implement the API it defines and provide the hardware capabilities it requires. This ensures that OCAP-certified DVR applications run on any OCAP-compliant DVR platform. Each OCAP DVR box has at least one tuner, one time-shift buffer per tuner, local storage for digital video playback/ recording, and local storage for a generalpurpose filesystem. As I described in “Inside Digital Video Recorders” (DDJ, July 2005), a tuner is used to obtain live content from the satellite or cable network and the time-shift buffer lets you perform trick operations (fast forward or pause, for instance) on the live content. CableLabs has also wisely differentiated file operations on a general-purpose filesystem (NTFS, ext2, ext3, and the like) from recording and playing back content to a storage device. On some DVRs, a general-purpose filesystem (GPFS) is used to store both content and data files. By contrast, other DVRs use a specialized filesystem for recording and playing back content. Typically, these filesystems are highly optimized for large block reads and writes, and shouldn’t be cluttered with small data files (some may not even let you create small files or use traditional file I/O APIs). Therefore, the OCAP DVR API offers APIs to detect if the medium is capable of general-purpose file I/O and content storage or retrieval (or both). In addition to these basic hardware features, every OCAP DVR is capable of recording live content, playing or watch-
Start of Movie
Start of Movie
Movie Credits (2:00)
Commercials
ing TV while recording, obtaining a listing of all available recordings, performing resource management, attaching permissions to content, and enforcing rights management and copy protection. Inverse Evolution The first wave of OCAP specifications tweaked existing European-based Multimedia Home Platform (MHP) Globally Executable MHP (GEM) specifications for North American cable products (http:// w w w . m h p . o rg / m h p _ t e c h n olo g y / other_mhp_documents/tam0927-pvr-pdrdvr-for-www-mhp-org.pps). By contrast, the initial iteration of the OCAP DVR spec was specifically designed for North American products and was not based on an existing European standard. At the same time, DVB was adding its own flavor of DVR functionality to GEM. Because both efforts were based on MHP and contained many common elements, CableLabs and MHP decided to merge the core functionality in both DVR working groups to create the shared standard “Digital Recording Extension to Globally Executable MHP” (MHP document number A088; see http://www.mhp.org/ mhp_technology/other_mhp_documents/ mhp_a088.zip). All A088 classes and interfaces are found in the org.ocap.shared namespace. However, if you try to create a Java-based DVR application using only A088, you’ll be bitterly disappointed when the compiler spews out an avalanche of errors when you compile it. This is because A088 provides common DVR interfaces and abstract classes that could be used on satellite, cable, or terrestrial products. It is missing key network and resource management classes needed to compile and link. CableLabs
End of Movie 2:15
Movie Credits (2:00)
End of Movie 2:15
Figure 1: Timelines map Normal Play Times to an OCAP DVR environment. In this illustration, even though commercials have been added to the content, the NPT still triggers at the appropriate time. 50
Dr. Dobb’s Journal, December 2005
and MHP intentionally avoided these topics because this functionality varies dramatically between European (DVB) and North American cable environments. Consequently, A088 must be supplemented with concrete classes to create a viable DVR solution. In OCAP, these concrete classes are located in the org.ocap.dvr namespace and are defined in the OCAP DVR spec. OCAP offers classes to handle features such as filesystems, resource, and network management. MHP’s implementation is found in the A087 spec (http://www.mhp.org/ mhp_technology/other_mhp_documents/mhp_a087.zip). It permits DVR applications to access European-oriented DVB Service Information (SI) and TVAnytime Java classes (TVAnytime defines features such as metadata to describe content). Storage Search Given that DVRs focus on playing back content, the first thing a DVR application typically does is detect the storage devices connected to the platform. This information resides in the StorageManager, a system object that monitors the availability of all storage-related devices. To obtain a listing of mounted filesystems, you call StorageManager.getStorageProxies( ). This returns an array of StorageProxy objects. StorageProxies implement a variety of interfaces, the most interesting of which are DetachableStorageOption, LogicalStorageVolume, and MediaStorageVolume. The DetachableStorageOption interface is implemented by a StorageProxy that can be hot-plugged (or dynamically added or removed). These type of StorageProxy objects will be found on DVR platforms that have IEEE 1394, SATA, or USB 2.0 port(s). It is important to test if the StorageProxy implements DetachableStorageOption before attempting file I/O on it because the filesystem may need to be mounted before it is used. The filesystem may be mounted by calling DetachableStorageOption.makeReady( ) and unmounted via DetachableStorageOption.makeDetachable( ). When a storage device is hot-plugged, the StorageManager generate events to all interested parties that have attached a listener via the addStorageManagerListener( ) method. These events are divided into three categories: • STORAGE_PROXY_ADDED. A storage device was added to the DVR. • STORAGE_PROXY_CHANGED. A storage device changed state (for example, the filesystem may be mounted or unmounted). • STORAGE_PROXY_REMOVED. A storage device was removed from the DVR. http://www.ddj.com
The second interface surfaced by StorageProxies is LogicalStorageVolume. Again, some DVR storage devices may not permit you to perform general-purpose file I/O on them. If the StorageProxy implements LogicalStorageVolume, then you can get path information via the LogicalStorageVolume.getPath( ) and set file attributes with LogicalStorageVolume.setFileAccessPermissions( ). The third critical interface exposed by a StorageProxy is MediaStorageVolume. Storage devices that host digital video and multimedia content implement the MediaStorageVolume interface. It offers methods to report total space available to record content and the maximum bitrate of content it can record or playback. This is vital information because you don’t want to record a 20-MBps High Definition stream onto a USB Thumb drive that is only capable of writing 11 MBps. Once you know all the active storage devices that can store content, the next step is to obtain a listing of available content. This is done by obtaining an instance of the RecordingManager from the RecordingManager.getInstance( ) method. For OCAP platforms, this returns an org.ocap.dvr.OcapRecordingManager object (DVB platforms will return a org.dvb.tvanytime.pvr.CRIDRecordingManager). The OcapRecordingManager has these responsibilities:
form trick operations such as pause, rewind, and fast-forward on live TV. Nestled within the RecordingRequest is a RecordedService. To obtain this service, you call RecordingRequest.getService( ). Once you have a RecordedService, you can play it with the Java Media Framework (JMF). Simply feed the locator returned from RecordedService.getMediaLocator( ) into Manager.createPlayer( ) and the content is presented like any other MPEG content. While you can use traditional JMF APIs to play DVR content, you can’t create a robust solution without taking advantage of OCAP’s DVR-specific JMF extensions. These enhancements are defined in the org.ocap.shared.media classes and enable your applications to obtain time-shift buffer
attributes, monitor timelines, and receive DVR-specific events. Time-shift buffers are circular buffers that a DVR uses to enable trick modes (see “Inside Digital Video Recorders” DDJ, July 2005). A TimeShiftControl represents a moving window in the overall content where you can perform trick operations on live content and it offers methods to query the size and starting and ending positions of the time-shift buffer. Timelines integrate Normal Play Times (NPT) into JMF’s media time concept (that is, the playback duration of the content). An NPT is a bookmark in the content and this bookmark will be valid no matter how the content is edited. For example, in Figure 1, an application wants to display a
• Managing recorded services. • Managing resources. • Scheduling recordings. • Resolving resource and recording conflicts. • Starting and stopping recordings. The first responsibility of the OcapRecordingManager is to maintain a list of recordings (or recorded services) that can be replayed. To obtain this listing of content, you call the OcapRecordingManager.getEntries( ) method. This returns a RecordingList object that can be navigated with a RecordingListIterator. When you arrive at the content you wish to play, you call the RecordingListIterator.getEntry( ) method to obtain a RecordingRequest object (for OCAP platforms this will be an OcapRecordingRequest). At first, it may seem confusing to have to navigate a list of RecordingRequest objects just to play content. However, DVR programming requires a change in mindset from playing traditional digital video files. Normally, in nonDVR environments, you must wait until the recording completes before you can play video files. By contrast, DVRs let you play content that is still actively being recorded. This is how DVR applications can perhttp://www.ddj.com
Dr. Dobb’s Journal, December 2005
51
menu on the screen exactly when the movie credits are being played (say, two hours into the movie). Even if the broadcaster inserts commercials into the content, the NPT triggers exactly when the credits are displayed (in this case, 150 minutes into the movie). When content is edited (that is, information is added or removed), an NPT discontinuity is generated and this discontinuity is represented by a single timeline. In OCAP, a JMF media time is the summation of all timelines in the content. In Figure 1, even though commercials have been added to the content, the NPT still triggers at the appropriate time. The third JMF enhancement provided by the OCAP DVR spec are additional events and exceptions to monitor the state of the time shift and timeline attributes. For instance, if you rewind to the beginning of the content, you will receive a BeginningOfContentEvent. Similarly, if you fast forward or play past the end of the content, you’ll get a RateChangeEvent. In addition, if you’re playing back live content and fast forward past the end, you will not only receive a RateChangeEvent, but you may also get an EnteringLiveModeEvent (see Listing One). EnteringLiveModeEvent lets you know that you forwarded past the end of the time-shift buffer and are now displaying live video at normal speed. Digital Video Recordings While playback supervision is a critical element of the RecordingManager’s functionality, its primary charter is to supervise recordings. The RecordingManager’s responsibilities involve scheduling recordings, resource management, and resolving conflicts. You initiate a recording by calling RecordingManager.record( ). This method takes one parameter, a RecordingSpec. A RecordingSpec is an abstract base class that describes the content you wish to record. Because it’s an abstract class, you must pass in a class that inherits from this class to record content. For example, if you wish to record the content that is currently being time shifted, you can use the ServiceContextRecordingSpec class. The ServiceContextRecordingSpec lets you specify when to start recording and how long
to record the content. Typically, your application would call this API if the user hits the record button while watching TV. Unfortunately, if the viewer decides to tune away from the current channel, the RecordingManager terminates any recording that was initiated by a ServiceContextRecordingSpec. If you want to ensure that
“Modern DVRs offer at least two tuners that let viewers make two simultaneous recordings” your recording isn’t aborted by a channel change, then you must use a ServiceRecordingSpec. A ServiceRecordingSpec lets the RecordingManager know that you want to record content associated with a specific channel (service) and is not tied to what program the viewer currently is watching at the time the recording is initiated. This type of RecordingSpec is useful when you want to schedule a recording ahead of time (say, if you want to record a specific football game or a concert). Resource Monitor First-generation cable and satellite DVRs had a single tuner (or source of content). This meant users couldn’t watch one program while recording another or simultaneously record two programs. More modern DVRs offer at least two tuners that let viewers make two simultaneous recordings or surf on one tuner while recording on another. Alas, although a dual-tuner product has fewer recording restrictions than a single-tuner solution, there are still limitations (for instance, they aren’t capable of three or more simultaneous recordings).
Listing One sample controllerUpdate() processing for DVR applications All JMF applications implement a controllerUpdate() method The OCAP DVR specification adds new events that DVR applications should listen to. In this illustration, the listener monitors EnteringLiveModeEvent
public synchronized void controllerUpdate(ControllerEvent event) { // this event will be received when the DVR JMF player is playing live // content from a tuner. Typically, this event will be received when // the user does a trick operation (such as a fast forward) that causes // the player to run out of recorded digital video and automatically
52
Conclusion The OpenCable Application Platform has long promised that it would break the stranglehold that proprietary platforms have on the U.S. cable market. However, since it didn’t offer the DVR capabilities that viewers crave, cable companies have been forced to use proprietary DVR solutions to satisfy consumer demand. Thankfully, the release of the OCAP Digital Video Record specification removes the last hindrance to widespread acceptance of OCAP. Finally we will be able to write Java applications to time-shift DVR content, record our favorite TV programs, and play these programs back with trick controls. Clearly, the release of this specification and boxes that will soon follow is a significant milestone in the evolution of interactive television. DDJ
// start playing live content. if (event instanceof EnteringLiveModeEvent) { // your application work would be done here... } // this event will be received when the DVR JMF player is // playing is playing recorded content. This typically // is generated when the application performs a trick operation // (i.e. pause or rewind) on live content.
import org.ocap.shared.media; import org.ocap.dvr; // // // // // //
Thus, no matter how many tuners are in the box, users will eventually request too many recordings and something must perform arbitration (that is, an object must decide which recording requests will be accepted or rejected based on hardware resources). This conflict resolution process is very network specific and consequently, it is not defined in org.ocap.dvr.shared. Rather, in an OCAP DVR, the OcapRecordingManager cooperates with the Multiple Service Operatorspecific monitor app to resolve conflicts. Recall that in “The OpenCable Application Platform” (DDJ, June 2004), I mentioned the monitor application has access to privileged network resources and is responsible for resolving all resource conflicts on the platform. The OCAP DVR spec extends the monitor application’s responsibilities to include resolution of DVR resource conflicts. For instance, if users have scheduled two recordings at 8:00 PM and an Emergency Alert System (EAS) is broadcast at 8:02 PM, the OcapRecordingManager alerts the monitor application of the conflict and the monitor application would then abort the lower priority recording to ensure that the EAS broadcast has access to the tuner so that the emergency alert could be broadcast.
}
else if (event instanceof LeavingLiveModeEvent) { // your application work would be done here... }
Dr. Dobb’s Journal, December 2005
DDJ http://www.ddj.com
XML-Binary Optimized Packaging XML and nontext data can work together ANDREY BUTOV
F
or several years now, the development community has followed the emergence of XML-based solutions to common data-representation problems. As a metalanguage, XML is a success. A cursory search reveals several of the more popular custom languages utilizing XML as a metalanguage, including MathML for representing mathematical concepts [1], Scalable Vector Graphics (SVG) for representing two- dimensional vector and raster graphics [2], and Really Simple Syndication (RSS) [3] for distributing news headlines and other relatively frequently updated data such as weblog postings, for consumption by, among other things, RSS aggregators. While most XML-derived languages are still used mostly for specialized representations of textual data, not all data domains are suitable for text-based representation. Consequently, there is a growing need to embed binary data into XML documents — in most cases, mixing binary data with plain-text XML tags for contextual description. While there is no standard for doing this as of yet, several approaches have had more success than others. In particular, SOAP [4] has long been used as a means of exchanging structured information, but there are numerous other proprietary implementations of technologies meant to overcome the binary-data dilemma. In January 2005, the World Wide Web Consortium released a document presenting the latest version of its recommendation of a convention called “XML-
Andrey is a software developer in the Fixed Income division of Goldman Sachs & Company. He can be contacted at andreybutov@ yahoo.com. http://www.ddj.com
binary Optimized Packaging” (XOP). The recommendation presents a way of achieving binary data inclusion within an XML document. There are several common arguments against pursuing the path laid out in the recommendation that are worthy of exploration; however, in its recommendation of XOP, the W3C presents a valid and interesting piece of technology worthy of examination. The Current State of Things The W3C’s XOP proposal stems from a long-standing need to encode binary data in XML documents, which are, in nature, text-based animals. Before exploring the W3C proposal itself, take a look at some of the problems surrounding the inclusion of binary data in XML documents, and how these problems are currently being circumvented. XML is usually presented as a way of describing text data within the context of a well-defined document containing metainformation (which is also text based) meant to bring some structure and form to that text data. There are, however, several domains that do not lend themselves nicely to being represented with textual data only. Most of these domains include the need to represent sound or video data in one form or another. In fact, the W3C itself has outlined several use cases that display a need to encode binary data in an XML document [5]. Assume that you want to somehow include Figure 1 in a transmission of an XML file describing a context in which the picture plays some role. You cannot simply place the binary data composing this picture in between two tags, as in Example 1(a), then transmit the XML file. The problem stems from the “text-centric” nature of XML. Binary data such as this can contain not only null characters (0x00 bytes), but can contain other string sequences, such as
the error because the XML file itself would no longer be properly formed; otherwise, the binary data itself could be improperly interpreted, causing a relatively easyto-spot problem of improper syntax turn into an issue dealing with data integrity (such as in the case of transmitting binary financial data rather than photographs).
“There are, in fact, several approaches to circumventing the binary inclusion problem” By the way, the 2-byte sequences previously mentioned are only two culprits. They are enough to make the point, but there are others. There are, in fact, several approaches to circumventing the binary inclusion problem. One popular (albeit naïve) approach is to place the binary segment into a tag [6]. Developers new to XML tend to think of this tag as a panacea, when in reality, using CDATA only mitigates the problem caused by directly embedding binary data between two arbitrary tags. In this case, while it is true that the data inside the CDATA section will not be validated as XML, CDATA sections cannot be nested, and thus, instead of crossing your fingers and hoping your binary data does not contain a null byte, you cross your finders and hope that the binary data does not contain the sequence ]]>. (Early in my career, I was part of a group of developers who used this exact method to implement a piece of technology responsible for delivering binary financial data inside an XML document via a TCP connection to be displayed in a Flash applet through a web browser. At the risk of nullifying my point, I do not remember ever having an incident of the 53
XML parser throwing a fit over an accidental ]]> sequence — but are you willing to take the chance with your application?) Another approach is to encode the binary data into some string representation. In fact, the XML Schema [7] defines the base64Binary type that can be used for this purpose, as in Example 1(b). This approach is widely used, not only because of the fact that it effectively circumvents the “bad bytes” problem of directly embedded binary data, but also because of the simplicity of implementation. There are many free and shareware utilities that can be used to Base64-encode data. It does, however, have its drawbacks — one issue dealing with
Figure 1: Cleo the puppy. (a)
Cleo Black q^@/0?%5
(b)
Cleo Black 6f9cd3e5(...)
(c)
Cleo Black http://www.somehost.com/cleo.jpg
Example 1: (a) Bad idea; (b) base64Binary encoding; (c) referencing the binary data. 54
space, and one dealing with time (and all this time you thought sacrificing one would bring benefits from the other). Encoding binary data in base64Binary typically produces output that is about one-third larger than the original binary data [8], and at the same time, it takes time to both encode the data at the source, and decode the data at the destination, although the cost there would depend on a number of factors. Yet another approach to the binary inclusion problem is to simply not include the binary data at all. Instead, a link (typically in the form of a URI reference) is presented instead of the binary data, while the data itself is kept at the referenced location. Example 1(c) presents an example of this. This approach has the advantage of not only bypassing all the issues of direct binary data embedding and encoding size/time performance hits, it also allows for the binary data itself to be modified at the referenced source without retransmitting the XML file containing the reference (although this could also be a bad thing, depending on context). Indeed, there are various current technologies that implement one or, more typically, a combination of these approaches to enable inclusion of binary data in XML documents. Probably the most common technology that addresses this issue is Simple Object Access Protocol (SOAP) [4], or more accurately, SOAP with Attachments (SwA) [9], which utilizes a system of references in conjunction with MIME to deal with binary data in XML. An example of another similar technology is Microsoft’s Direct Internet Message Encapsulation (DIME) [10]. Both of these technologies warrant pages (books?) of additional description, and fall outside the scope of this article. W3C Recommendation The first document in the chain of XOPrelated publications was a Working Draft published in February 2004. Public comments on the first W3C Working Draft of XOP were requested at the time of publication [11] and a W3C public archive of the related messages is available for browsing at http://lists.w3.org/Archives/Public/ xml-dist-app/. Subsequent comments on the topic resulted in one more Working Draft [12], a Candidate Recommendation [13], a Proposed Recommendation [14], and finally, the W3C Recommendation serving as the topic of this article [15]. The recommended convention proposes a more efficient way to package XML documents for purposes of serialization and transmission. An XML document is placed inside an XOP package (see Figure 2). Any and all portions of the XML document that are Base64 encoded are extracted and optimized. For each of these chunks of extracted and optimized data, Dr. Dobb’s Journal, December 2005
an xop::Include element is placed to reference its new location in the package. The recommendation makes a note that the encoding and eventual decoding of the data can be part of managing the XOP package — meaning that upon encoding, the internal links are automatically generated, and upon extraction, the links are automatically resolved, and the data can even be presented in its original intended format (sound, picture, and so on [15]). At its core, the recommendation proposes a rather simple idea. Binary data can be included alongside plain-text XML data, and the XOP package processing would take care of the optimization (which promises to result in a much smaller dataset than the equivalent Base64encoded data), the internal references would be resolved, and all levels of software above the XOP package management would not have to worry about managing the binary data either on the encoding or the decoding side. An additional benefit, which I believe would drive this technology forward in comparison with DIME or SwA, is that existing XML technologies would continue to work with the XOP package data. This includes already popular XML-based technologies such as XQuery [16] for managing collections of data, XSLT [17] for content transformation, and even XML Encryption. In fact, XML Encryption [18] is referenced in the XOP recommendation itself as the solution concerning issues of security in XOP. Community Feedback The W3C recommendation (as well as its predecessors) raised some common arguments in the development community, including one stemming from a misunderstanding of the recommendation: • What the W3C recommendation is NOT. The recommendation does not propose a binary format for XML documents. Although the W3C XML Binary Characterization Working Group does exist for the purpose of exploring the benefits of forming a binary protocol for XML, the recommendation of XOP focuses on a different issue entirely. XOP is merely a way of serializing XML in a way that would make it more efficient to include Base64-encoded data. • What about the purity of XML? This argument is akin to people arguing that ANSI C ruined the purity of K&R C. A programming language (and in this case, metalanguage) remains alive partially due to its ability to adapt to the needs of its users. Any piece of technology that remains too rigid to evolve with those needs eventually dries up. It is precisely the flexibility of XML that has put it in the place it stands today. http://www.ddj.com
• What about a simpler solution? Gzip anyone? Using gzip or some other compression scheme on XML data only has the potential of mitigating the size of the data. However, with this approach, the data is no longer human/parser readable, as would be the case with XOP, where only the Base64-encoded data is optimized and stored away, leaving the rest of the content of the XML well formed, self describing (in as much as XML is self describing), and in a way still able to be manipulated by various XML-focused tools. A compression solution would also not allow large chunks of binary data to be placed at the end of the file, leaving the parser with no choice but to deal with the entire data set upon decompression. • Optimization of Base64 types only? This is actually a valid point. The recommendation does imply that XOP is currently enabled to only optimize Base64encoded data. If the recommendation does evolve into some sort of a formal specification, and the development community picks it up as a practical solution, it would not be at all surprising if the demand grows for XOP to remove this limitation. This remains to be seen. Conclusion The existence of SwA, DIME, and countless proprietary technologies is evidence that there is a need to solve a relatively long-standing problem in the development community. The W3C Recommendation attempts to address this problem in a way that would put the burden of dealing with binary data inclusion into the specification of XOP rather than keep it at the application level, which in the long run results in various proprietary solutions, such as the case with SwA and DIME, and produces many forked roads. XML-binary Optimized Packaging is a sound, well thought-out idea that, in spite of common arguments, deserves consideration. References [1] David Carlisle, Patrick Ion, Robert Miner, and Nico Poppelier, Editors. Mathematical Markup Language (MathML) Version 2.0 (Second Edition). World Wide Web Consortium, W3C Recommendation, October 21, 2003. http:// www.w3.org/TR/MathML2/. [2] Jon Ferraiolo, Fujisawa Jun, and Dean Jackson, Editors. Scalable Vector Graphics (SVG) 1.1 Specification. World Wide Web Consortium, W3C Recommendation, January 14, 2003. http://www.w3 .org/TR/SVG11/. [3] Gabe Beged-Dov, Dan Brickley, Rael Dornfest, Ian Davis, Leigh Dodds, Jonathan Eisenzopf, David Galbraith, R.V. Guha, Ken MacLeod, Eric Miller, http://www.ddj.com
Original Infoset
Extraction
XOP Infoset + Extracted Content
Reconstitution
Reconstituted Infoset
Serialization/Deserialization XOP Package XOP Document Extracted Content
Figure 2: XOP Overview. Aaron Swartz, and Eric van der Vlist. RDF Site Summary (RSS) 1.0, 2001., http://web.resource.org/rss/1.0/spec. [4] Martin Gudgin, Marc Hadley, Noah Mendelsohn, Jean-Jacques Moreau, and Henrik Frystyk Nielsen, Editors. SOAP Version 1.2 Part 1: Messaging Framework. World Wide Web Consortium, W3C Recommendation, June 24, 2003. http://www.w3.org/TR/soap12. [5] Mike Cokus and Santiago PericasGeertsen, Editors. XML Binary Characterization Use Cases. World Wide Web Consortium, W3C Working Draft, February 24, 2005. http://www.w3.org/TR/ xbc-use-cases/. [6] Tim Bray, Jean Paoli, C.M. SperbergMcQueen, Eve Maler, and Frangois Yergeau, Editors. Extensible Markup Language (XML) 1.0 (Third Edition). World Wide Web Consortium, W3C Recommendation, February 4, 2004. http:// www.w3.org/TR/REC-xml/. [7] Paul V. Biron, Kaiser Permanente, and Ashok Malhotra, Editors. XML Schema Part 2: Datatypes Second Edition. World Wide Web Consortium, W3C Recommendation, October 28, 2004. http:// www.w3.org/TR/2004/REC-xmlschema220041028/datatypes.html#base64Binary. [8] Sameer Tyagi. “Patterns and Strategies for Building Document-Based Web Services.” Sun Technical Articles, September 2004. http://java.sun.com/developer/ technicalArticles/xml/jaxrpcpatterns/ index.html. [9] John J. Barton, Satish Thatte, and Henrik Frystyk Nielsen, Editors. SOAP Messages with Attachments. World Wide Web Consortium, W3C Note, December 11, 2000. http://www.w3.org/TR/ SOAP-attachments. [10] Jeannine Hall Gailey. “Sending Files, Attachments, and SOAP Messages Via Direct Internet Message Encapsulation,” MSDN Magazine, December 2002. http://msdn.microsoft.com/msdnmag/ issues/02/12/DIME/default.aspx. [11] Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Dr. Dobb’s Journal, December 2005
Optimized Packaging. World Wide Web Consortium, W3C Working Draft, February 9, 2004. http://www.w3.org/ TR/2004/WD-xop10-20040209/. [12] Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Optimized Packaging. World Wide Web Consortium, W3C Working Draft, June 8, 2004. http://www.w3.org/TR/2004/ WD-xop10-20040608/. [13] Martin Gudgin, Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Optimized Packaging. World Wide Web Consortium, W3C Candidate Recommendation, August 26, 2004. http://www.w3.org/TR/ 2004/CR-xop10-20040826/. [14] Martin Gudgin, Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-Binary Optimized Packaging. World Wide Web Consortium, W3C Proposed Recommendation, November 16, 2004. http://www.w3.org/ TR/2004/PR-xop10-20041116/. [15] Martin Gudgin, Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Optimized Packaging. World Wide Web Consortium, W3C Recommendation, January 25, 2005. http://www.w3.org/TR/xop10/. [16] Scott Boag, Don Chamberlin, Mary F. Fernandez, Daniela Florescu, Jonathan Robie, and Jerome Simeon, Editors. XQuery 1.0: An XML Query Language. World Wide Web Consortium, W3C Working Draft, February 11, 2005. http:// www.w3.org/TR/xquery/. [17] James Clark. Editor. XSL Transformations (XSLT) Version 1.0. World Wide Web Consortium, W3C Recommendation, 16 November 1999. http://www .w3.org/TR/xslt. [18] Joseph Reagle. XML Encryption Requirements. World Wide Web Consortium. W3C Note, March 4, 2002. http:// www.w3.org/TR/xml-encryption-req.
DDJ 55
A Mac Text Editor Migrates to Intel Porting a commercial Mac application TOM THOMPSON
I
n June of this year, Apple Macintosh developers discovered that they had a formidable challenge set before them. The bad news was that to continue in the Macintosh software market, they were going to have to migrate their Power PCbased applications to an Intel x86-based Macintosh. The good news was that the operating system, Mac OS X, had already been ported and was running on prototype x86-based systems. Furthermore, Apple Computer offered cross- compiler Xcode tools that would take an application’s existing source code and generate a “universal binary” file of the application. The universal binary file contains both Power PC and x86 versions of the program, and would therefore execute on both the old and new Mac platforms. (For more details on how Apple has engineered this transition, see my article “The Mac’s Move to Intel,” DDJ, October 2005.) Apple provided developer reports that describe the migration of an existing Mac to the new platform as relatively quick and easy. I don’t dispute those reports. There are situations where a particular application’s software design dovetails nicely with the target platform’s software. For example, if the application is written to use Cocoa, an object-oriented API whose hardware-independent frameworks implement many of Mac OS X’s system services, the job is straightforward. However, not all Mac applications are in such an ideal position. Many commercial Mac applications can trace their roots back to when the Mac OS API was the only choice (Mac OS X offers four), and Tom is a technical writer providing support for J2ME wireless technologies at KPI Consulting Inc. He can be contacted at
[email protected]. 56
most code was written in a procedural language, instead of an object-oriented one. When migrating an application to a new platform, developers are loath to discard application code that’s field-tested and proven to work. Using such “legacy” code potentially reduces the cost and time to port an application because the process starts with stable software. The downside to this strategy is that code idiosyncrasies or subtle differences in the implementation of the legacy APIs on the new platform may hamper the porting process. So, for the majority of Mac developers, the move to Intel might look more like a leap of faith than an easy transition. This brings us to the $64,000 (or more) question: For most existing Mac applications, how difficult and costly is it to migrate a Power PC Mac application to the x86-based Mac platform? To see where the truth lies, I present in this article a case study of the porting process for a commercial Mac application. An Editor with Rich Features and History The application in question is Bare Bones Software’s BBEdit, an industrial-strength text editor (http://www.barebones.com/). “Text editor” is perhaps a bit of a misnomer here, because over the years, BBEdit’s text-processing capabilities have evolved to suit the needs of several categories of user. Besides being a powerful programming editor for writing application code, web wizards use it to write HTML web pages, while professional writers use BBEdit to produce all sorts of text documents, ranging from manuals to novels. BBEdit offers a wide variety of features and services to meet the demands of these three disparate groups of users. For programming work, the editor displays the source code of C, C++, Objective-C, Java, Perl, Python — in total, about 20 programming languages with syntax coloring. It features integration with the Absoft tools and interoperates with the Xcode tools as an external editor. For HTML coding, menu choices let you quickly generate HTML tags. The HTML code is also syntax colored, and syntax checks can be run on the Dr. Dobb’s Journal, December 2005
HTML to spot coding errors. Furthermore, BBEdit can render the HTML that you just wrote so that you can preview the results immediately and make changes. When the web page’s code is ready, BBEdit lets you save it to the server using an FTP/SFTP connection. For just plain writing, the editor is fast, allowing you to quickly search, change, and scroll through large documents
“BBEdit draws upon many of the APIs available in Mac OS X to implement its feature set” consisting of 7 million words or larger. A built- in spellchecker lets everyone — programmers, web wizards, and writers alike — check for spelling errors. Finally, it provides a plug-in interface so that new features and tools can be added to the application in the future. The current 8.2 version of BBEdit, while changed in many ways from the original version (the World Wide Web didn’t even exist when the first version of BBEdit was written), still owes a lot to the code design of its ancestors. To appreciate how the Bare Bones Software team managed the current transition to the x86 platform, it helps to review the editor’s rich and complex history. This is due to the fact that the program has been revised numerous times and completely rewritten several times. As you’ll see, certain design decisions, some made years ago, profoundly affected the current (and third) transition to the x86 platform. BBEdit is a classic Mac application in every sense of the word, being 15 years old. It was written in 1990 to provide a lightweight — yet fast — code editor for the then-extant 68K Mac platform. The bulk of BBEdit was originally written in Object Pascal, while performance-critical sections of the program were written in C and 68K http://www.ddj.com
assembly. Mac old-timers will recall that the preferred programming language for Mac applications was Pascal, and that you accessed the Mac APIs with Pascal stackbased calling conventions. C compilers of that era used glue routines to massage the arguments of a C-based API call so that they resembled a Pascal stack call. The first major code change BBEdit underwent was when Apple transitioned the Mac platform from the Motorola 68K processor to the Motorola-IBM Power PC processor in 1994. The software engineers considered what was required to port the program, and the decision was made to revamp portions of the editor, in particular using C to replace the 68K assembly code. During the 1999 –2000 timeframe, BBEdit was rewritten entirely in C++. The reason for the programming language change was that since the early ’90s, Pascal was on the wane, as many tool vendors and universities embraced C and C++ as the programming languages of choice. The shift to C++ was made due to the dwindling support for Pascal compilers. The compilers of that transition era (notably Metrowerks’ CodeWarrior) were able to take an application’s source code and generate a “fat binary,” which was an application that contained both 68K code and Power PC code. Such an application, while larger in size, could run at native speeds on either a 68K- or Power PCbased platform during the transition period. The current universal binary scheme is similar in concept. The next code change occurred when BBEdit was migrated from Mac OS 9 (which is considered the “Classic” environment today) to Mac OS X. Work began in 1999, and was completed by Mac OS X’s launch in 2001. Although the new OS offered several different APIs (Cocoa, POSIX UNIX, and Java), Apple had to support the Classic APIs, or else the platform would lose a huge body of existing Mac OS software. To this end, Apple provided a fourth set of APIs, the Carbon APIs. On the front end, Carbon resembles the Classic Mac OS APIs. On the back end, Carbon provided existing applications access to all of Mac OS X’s system services, such as memory protection and preemptive multitasking. However, Carbon lacked a handful of the Classic APIs that caused compatibility problems, and some of the capabilities of the retained APIs were modified slightly so that they would work within Mac OS X’s frameworks. For the migration to Mac OS X, C, C++, and Carbon were used to handle the core application services, such as text manipulation. However, where deemed appropriate, the other Mac OS X APIs were used to implement specific application features. For example, Cocoa provided access to the Webkit framework, which renders the http://www.ddj.com
HTML for BBEdit. These sections of the application were thus written in Objective-C and Objective-C++. Other services, such as SFTP, utilized the POSIX UNIX API. Table 1 shows how BBEdit draws on different Mac OS X APIs to implement its features.
For the move to Mac OS X, a complete rewrite of BBEdit to Cocoa was considered. Given the company’s finite resources, it was decided that BBEdit’s customers would be best served by adding features that they wanted, rather than rewriting the
Endian Issues
T
he infamous “Endian issue” occurs because the Power PC and Intel processors arrange the byte order of their data differently in memory. The Power PC processor is “Big Endian” in that it stores a data value’s MSB in the lowest byte address, while the Intel is “Little Endian” because it places the value’s LSB in the lowest byte address. There is no technical advantage to either byte-ordering scheme. When you access data by its natural size (say using a 16-bit access for a 16-bit variable), how the bytes are arranged doesn’t matter and programs operate as they should. However, when you access multibyte variables through overlapping unions, or access data structures with hard-coded offsets, then the position of bytes within the variable do matter. Although the program’s code executes flawlessly, the retrieved data is garbled because of where the processor stores the bytes in memory. Stated another way, code written for a Big-Endian processor accesses the wrong memory locations on
a Little-Endian processor. Consequently, the wrong values are fetched, which produces disastrous results. The Endian issue manifests itself another way when multibyte data is accessed piecewise as bytes and then reassembled into larger data variables or structures. This often occurs when an application reads data from a file or through a network. The bad news is that any application that performs disk I/O on binary data (such as reading a 16-bit audio stream), or uses network communications (such as e-mail or a web browser), can be plagued by this problem. The good news is that each set of Mac OS X APIs provide built-in methods that can reorganize the bytes (this rearrangement is termed “byte swapping”) for you. For more information on these routines, see the Apple Core Endian Reference document (http:// developer.apple.com/documentation/ Carbon/Reference/CoreEndianReference/ index.html). —T.T.
Best Programming Practice Rules 1. Follow the guidelines. Apple provides documents that describe safe coding practices for both Mac OS X and for tailoring applications to be universal binaries. You’ll save yourself a lot of effort by studying and following the coding recommendations that these documents provide. 2. No warning messages discipline. When you build your application, the compiler shouldn’t generate warning messages. While some of the warnings may seem trivial, they also hint of trouble lurking in the code. All problematic code should be examined carefully and revised to eliminate the warning messages. The BBEdit engineers have a strict policy on not allowing any warning messages during a code build. 3. When compiling, more is better. Like a patient seeking a second opinion for a subtle malady, often a developer will seek a second opinion on the quality of an application’s code
Dr. Dobb’s Journal, December 2005
by building it with a second (and even third) compiler. The other compiler, through its warning messages, can point out subtle coding problems overlooked by the first compiler. Rich Siegel puts it concisely: “More compilers result in higher quality code.” 4. Examine the code, line by line. This is the hardest process to follow, but it is probably the most important. During each migration of BBEdit, the engineers scrutinized every line of code in the application. Such careful examination of the code not only identifies what code needs to be revised to support the port, but it also discovers subtle code issues not uncovered by rules two and three. It’s also valuable for winnowing code whose purpose has been taken over by a new API or system service. The change in BBEdit to save its preferences using an API rather than a custom resource was the result of this examination. —T.T.
57
application. According to Rich Siegel, BBEdit’s creator, “New capabilities will be added to the editor as needed, and in doing so, we’ll use the right tool for the right job.” In other words, if the Carbon APIs offered the best way to implement the fea-
tures, then the feature code would use the Carbon APIs. The Final Frontier: The Shift to Intel As has become obvious, BBEdit draws upon many of the APIs available in Mac
Feature/Function
API Used
Programming Language
Editor Screen display FTP transfer OSA scripting Webkit framework Spelling framework SFTP Authentication UNIX scripting
Carbon Carbon Carbon Carbon Cocoa Cocoa POSIX POSIX POSIX
C/C++ C/C++ C/C++ C/C++ Objective-C/Objective-C++ Objective-C/Objective-C++ C/C++ C/C++ C/C++
Table 1: Mac OS X APIs that BBEdit uses to provide features.
58
Dr. Dobb’s Journal, December 2005
OS X to implement its feature set. The result is that the application consists of an amalgam of C/C++ code and ObjectiveC/Objective-C++ code. On the surface, this mix of APIs and programming languages might complicate the shift to the x86 platform. However, the earlier port of BBEdit to the Power PC version of Mac OS X worked in BBEdit’s favor here, because it limited the problems the team had to deal with to just those brought about by the OS’s behavior on the new platform. First, with Mac OS X 10.4, the OS itself was no longer a moving target. In earlier releases of the OS, its underlying architecture — notably the kernel APIs — was in a state of flux and subject to change. And change these low-level APIs they did, as Apple refined the kernel and underlying frameworks to make improvements. As a consequence, each new release of the OS left broken applications in its wake, an unpleasant outcome that dissuaded many Mac users from switching to Mac OS X. Mac users, after all, want to get work done, not futz with the software. In 10.4, the kernel programming interfaces (KPIs) have been frozen, and a versioning mechanism lets drivers and other low-level software handle those situations when the KPIs are changed to support new features. The result is an underlying infrastructure for the OS that’s stable and consistent across different platforms. This, in turn, makes the porting process manageable. According to the BBEdit engineers, Mac OS X 10.4 does a good job at hiding the hardware details, while still providing lowlevel services (such as disk I/O). In addition, its APIs are mostly platform neutral, which means no special code is required to counter side-effects when invoking the APIs on each platform. Put another way, the code to call an API is identical for both platforms, and the results of the API call are identical; no glue code is necessary. The one side-effect of the x86 processor that Mac OS X can’t counter is the Endian issue. The Endian issue arises because of how the data bytes that represent larger values (such as 16- and 32-bit integers, plus floating-point numbers) are organized in memory. (For details, see the accompanying textbox entitled “Endian Issues.”) According to Siegel, the Endian issues “were minor, but present.” An earlier design decision actually side-stepped what could have been a major problem for the editor in this area. In 2000, BBEdit 6.0 began using Unicode to support the manipulation of multibyte text for Asian, Arabic, and certain European languages. Unicode is a standard for encoding complex language characters and describing their presentation. For example, Unicode not only specifies the characters in Arabic text, it http://www.ddj.com
also specifies an algorithm used to handle its bidirectional display. In turn, Unicode uses the 16-bit Unicode Transformation Format (UTF-16), which encodes the characters as 16-bit values. (For the record, there are variations of UTF used to encode 8- and 32-bit values.) UTF-16 uses three encoding schemes to describe the byte-ordering of the data: UTF-16BE, UTF-16LE, and UTF-16. In the first two schemes, the name describes the byte ordering, where BE represents BigEndian and LE represents Little-Endian. For the third encoding, a two-byte sequence at the start of the file makes up a Byte Order Mark (BOM) that describes the Endian encoding scheme. BBEdit 8.0, which was released in late 2004, uses the full Unicode conversion and rendering features of Mac OS X. These APIs automatically read a file’s encoding scheme and manage the data transfers and file I/O appropriately. By choosing to use Unicode early on, the Bare Bones Software engineers not only expanded the number of languages the editor could support, but also avoided what could have been a serious problem with reading and writing files when migrating to the x86based Mac platform. Another key revision made in BBEdit 8.0’s code design was that the application began using Mac OS X’s Preference Services API, rather than storing binary data in a custom resource. This modification also side-stepped the Endian problem. The biggest impact of the Endian issues was in BBEdit’s UI, particularly its menus. The menu choices were stored as tabular data in a custom resource. A byteswapping routine had to be written to massage the menu data so that it was properly organized in memory for the x86 platform. Interestingly, the plug-in mechanism was unaffected by the Endian issue. Since this mechanism passes data structures between methods, and these methods access the structures at their native data sizes, the order of the bytes didn’t matter. However, older plug-in modules that used the PEF binary format had to be recompiled into the Mach-o format, as that is the executable binary format both Mac OS X platforms will use going forward. The only code compatibility issue that the BBEdit engineers had to fix was that Apple made a data structure in the Carbon Alias Manager (which is used to locate files by name only on a local drive) opaque. That is, the data structure’s contents could no longer be accessed directly. Specifically, the userType field (used to specify the file’s type) in an AliasRecord was made opaque. Rather than access that structure directly, you now use getter and setter functions to manage this information. http://www.ddj.com
Lessons Learned It took the BBEdit engineers about 24 hours to make the changes described here so that BBEdit executed on the x86-based Mac platform. What took much longer was the regression testing that thoroughly stressed every feature of the application to ensure that subtle side-effects due to Endian issues or the APIs weren’t overlooked. This rigorous testing also had to verify that any changes to the code to facilitate the port didn’t adversely affect the execution of the Power PC version of the application. In all, testing took several weeks. At the end of this testing, Bare Bones Software announced BBEdit 8.2.3, which is now a universal binary. BBEdit is a sophisticated application that has survived several platform ports over the
Dr. Dobb’s Journal, December 2005
course of a decade. Despite its complex internal workings and equally complex history, the BBEdit port went smoothly. Solid code design was the key factor that made the code port a success. See the accompanying textbox entitled “Best Programming Practice Rules” for a list of good programming practices that BBEdit’s engineers followed to ensure the writing of quality code. Siegel summarizes the situation: “There were no unexpected bumps during the migration process, particularly when you consider we were using prerelease tools on a prerelease OS.” Bare Bones Software’s porting experience augers well for the migration of other Power PC-based Mac applications to the x86-based Mac platform. DDJ
59
SUMMER OF CODE
Google’s Summer of Code G
oogle’s Summer of Code was a unique and exciting program in which student programmers were provided stipends for creating new open-source projects or helping established ones. Over the summer of 2005, Google funded more than 400 projects to the tune of $5000 each, with $4500 going to the student and $500 to a mentoring organization. DDJ will be profiling some of the student participants over the coming months. Google’s open-source programs manager Chris DiBona and engineering manager Greg Stein led the Summer of Code project. DDJ recently talked to DiBona about the program. DDJ: What was the original goal of the Summer of Code? CD: The original impetus behind the program was to ensure that budding computer scientists wouldn’t let their programming skills diminish over the summer while working in a noncomputerrelated job. We thought that if we could make it possible for these students to work with the open-source community then they would be exposed to a whole new, very real, class of problem. This would then lead to more open-source software developers, programs, and better developers overall. DDJ: Did the final results meet your expectations? CD: From the very beginning the students far exceeded Google’s and my personal expectations. The quality of the applications alone caused us to double the number of accepted students from 200 to 419, and I think that easily a thousand of them proposed acceptable applications. Now that the program is over, the early results are pretty terrific, showing around 80 percent of the students having succeeded to execute on their projects to their mentors’ satisfaction.
DDJ: What was the biggest surprise coming out of SoC? CD: Just how advanced some of the projects ended up being. I remember thinking when I saw some of the projects that there was no way someone new to a project could pull them off. One, a CIL back end for GCC, which allows for the creation of CLR code from any GCC front-end language, should be preposterously difficult to do, but the student not only completed it, but did it in such a way that amazed his mentor, Miguel De Icaza. DDJ: What was the geographic distribution of participants? CD: We had 419 students taking part in the program from 49 countries. In the U.S. alone, we had students from 38 states. DDJ: Are they representative of open source as they are today, or are they signs of things to come? CD: I think that they are a little bit of the present and a big part of the future. Open source can be a little intimidating for the newcomer, and I think the Summer of Code helped to mix things up a bit and keep things fresh. Happily, a good number of the students have indicated that they intend to continue working on their open-source projects. DDJ: Where does SoC go from here? CD: Into the Fall, of course! We’re going to examine the feedback and make sure that the program was successful; if so, we may do another one next year. DDJ
Apache Axis2 JMX Front
A
pache Axis2 is a highly extensible Java-based web-service engine. Its extensibility comes mainly from the handler chain-based architecture. Axis2 allows configuring these handlers and other feathers mainly using XML files. There was no proper way to configure these settings while Axis2 was running in servers. The
goal of Axis2 JMX Front is to provide a JMX management interface for monitoring and configuring Axis2 at runtime. Axis2 JMX Front consists of a management class (MBean) named Axis2Manager, which provides access to all configurable modules. It handles everything regarding configuring various modules and
Name: Chathura C. Ekanayake Contact:
[email protected] School: University of Moratuwa, Sri Lanka Major: Computer Science and Engineering Project: Axis2 JMX Front Project Page: http://wiki.apache.org/ws/SummerOfCode/2005/JMXFront/ Mentors: Deepal Jayasinghe and Srinath Perera Mentoring Organization: Apache Software Foundation (http://www.apache.org/)
60
Dr. Dobb’s Journal, December 2005
Chathura C. Ekanayake provides a simple interface. This MBean has the functionality to configure settings of handlers, transport protocol handlers, and deployed services. For example, administrators can use this interface to turn http://www.ddj.com
CL-GODB: A Common Lisp GO Database Manipulation Library
C
L-GODB is a new interface to the GO Database (http://www.geneontology .org/) written in Common Lisp. The Gene Ontology (GO) is a collection of terms organized in a taxonomy representing a controlled vocabulary used to describe genes, gene products, their functions, and the processes they are involved in for a variety of organisms. The GO Database (GODB) represents the ontological information and gene product annotations in a convenient relational database format (the GO database uses MySQL). Until now, there have been no interfaces to the database that use Common Lisp. This is inconvenient as there are Bioinformatics and Systems Biology tools that employ the language (BioLingua, GOALIE, and the BioCYC suite, for instance). GOALIE, developed by Marco Antoniotti and Bud Mishra in NYU’s Bioinformatics Group (http://bioinformatics.nyu.edu/ ~marcoxa/work/GOALIE/) analyzes time
course data from micro-array clustering experiments. The CL-GODB library will be integrated into GOALIE, improving the tool’s functionality and efficiency. The library works by building an incremental, as-needed, internal image of the GO database contents in core. This improves the speed of queries and facilitates the construction of more complex predicates that may be needed in an application such as GOALIE. Users start by creating a handle that identifies their session and is linked to several hash indexes used in the in- core caching. Once they have connected to their copy of the GO database, they have access to a variety of built-in SQL queries, which take advantage of the indexing and add to the stored data. The queries range from getting basic information about a term, to finding a term’s lineage using a choice of hierarchies. As a testbed for the CL-GODB library, we built a GUI application that is available as a standalone executable. The CLGODB Viewer lets users browse the hierarchy with a graphical tree view and provides information about each term and
Samantha Kleinberg its associated genes, in a manner similar to that of several other GO viewer applications available online. Creating the CL-GODB was challenging at times, as it was my first project in Common Lisp. The biggest hurdle was making sure that case-sensitivity vagaries were taken care of, as Common Lisp and MySQL behave differently under Windows and UNIX. In the end, it did work and I learned more about the intricacies of SQL syntax than I ever wanted to know. DDJ
Name: Samantha Kleinberg Contact:
[email protected] School: New York University Major: Physics and Computer Science Project: CL-GODB Project Page: http://common-lisp.net/project/cl-godb/ Mentor: Marco Antoniotti Mentoring Organization: LispNYC (http://www.lispnyc.org/)
Figure 1: The CL-GODB user interface.
// Create Object String myObjectName = "Axis2:type=management.MyObject"; MyObject myObject = new MyObject(); // Register myObject using JMXManager JMXManager jmxManager = JMXManager.getJMXManager(); jmxManager.registerMBean(myObject, myObjectName); Example 1: Using JMXManager. off selected operations from web services, after they are deployed. This MBean is registered in an MBeanServer and published in a JMXConnectorServer. Remote management applications (JConsole, JManage, and so on) can access this MBean using RMI and call any function it provides. Therefore, administrators of Axis2 can log on to this interface to monitor and configure the system while it is running in servers. http://www.ddj.com
They can also manage different Axis2 engines running in different servers as a collection (cluster) using this interface. Axis2 JMX Front can be extended seamlessly with additional management functionality. Developers can add functions to the existing MBean or create separate MBeans without altering the rest of the code. After implementing a class with the required management functionality, they can call the Dr. Dobb’s Journal, December 2005
methods of the JMXManager class to register and publish objects of those classes as MBeans. Example 1 illustrates the use of JMXManager for registering a normal Java object named “myObject” as an MBean. Axis2JMX Front uses the Apache commons.modeler package for registering MBeans. Therefore, MBean developers are not required to provide a separate interface for their management objects. JMX Front loads all the JMX-specific classes at runtime to make the Axis2 build independent of JMX libraries. It also provides a separate class named “JMXAdmin” to handle all JMX-related features. Axis2 engine can load this class at runtime to JMX-enable the system. This allows Axis2 JMX Front to be deployed as an optional package, which can be integrated to Axis2 at deploy time. DDJ 61
Wide Character Support in NetBSD Curses Library
T
he current NetBSD curses library doesn’t support wide characters, which limits the use of NetBSD in countries with wide- character locales. The “Wide Character Support in curses” project adds wide-character support to the NetBSD curses library, complying with the X/Open Curses Reference to provide internationalization and localization. The difficulty of adding wide-character support to NetBSD curses lies in its internal character storage data structure and related functions, which assume an 8-bit character in each display cell. Adding wide-character support means adding a new character storage data structure to hold wide-character information. This structure holds not only the character but also the attributes, including any nonspacing characters associated with the display cell. The internal character storage data structure adds two linked lists for foreground/background nonspacing characters and uses spare bits in the attribute field for the character width, which are required for multicolumn characters. There is one storage cell per column, but the width fields are set differently for a multicolumn character. For an m-column-wide character, the first cell holds the width of the character, and the other m–1 cells hold the position information in their width fields. This offset is negative, making it easy to detect a cell belonging to a multicolumn character. To read a wide character from a keyboard, a distinction must be made between a function key sequence and a wide-character sequence. The keymap routines for narrow character input are used to detect function keys, and the stateful wide-character conversion routine mbrtowc( ) is used to assemble input bytes into a valid wide character.
Some existing narrow character routines have been modified to work with wide Ruibiao Qiu characters. The new storage data structure makes screen-refreshing code more complicated because the NetBSD curses library uses a hash function to determine if a screen needs to be refreshed. For wide-character support, the hash function must include the nonspacing characters as well to capture the changes in rendition. Another issue is when a character is added or deleted, a check must be made to detect if that character was part of a multicolumn character. All parts of the multicolumn character are removed in this case. The modified curses library was tested with three widecharacter locales — Simplified Chinese, Traditional Chinese, and Japanese. Test results show that twice the memory is generally required to support wide characters. DDJ Name: Ruibiao Qiu Contact:
[email protected] School: Washington University Major: Doctoral Candidate, Computer Science and Engineering Project: Wide Character Support in Curses Project Page: http://netbsd-soc.sourceforge.net/projects/wcurses/ Mentors: Julian Coleman and Brett Lymn Mentoring Organization: The NetBSD Project (http://www.netbsd.org/)
gjournal: FreeBSD GEOM Journaling Layer
T
he aim of the gjournal project is to create a data journaling layer for FreeBSD’s GEOM storage device layer. The idea of gjournal was born from the observation that FreeBSD doesn’t currently have a journaling filesystem, but in an early phase the specification was extended to include copy-on-write (COW) functionality. The GEOM subsystem is a modern kernel-based framework that manages pretty much all aspects of usage and control of storage devices. It’s based on the concept of classes. A GEOM class can be a source of data or it can implement data transformations in a completely transparent way. All classes can be arbitrary combined in a hierarchy in the form of a directed acyclic graph. Examples of existing GEOM classes are gmirror, which consumes two or more underlying class instances (called “geoms”) and provides one that duplicates and distributes I/O requests to them (a RAID 1 layer); and geom_dev, which consumes all disk device geoms and creates entries in the /dev filesystem hierarchy for them. The gjournal is implemented as a GEOM class that consumes two geoms and produces one. The first of the two consumed geName: Ivan Voras Contact:
[email protected] School: University of Zagreb Major: Electrical Engineering and Computing Project: gjournal Project Page: http://wikitest.freebsd.org/moin.cgi/gjournal/ Mentors: Pawel Jakub Dawidek and Poul-Henning Kamp Mentoring Organization: The FreeBSD Project (http://www.freebsd.org/)
62
oms is designated as a “data device” and the second as a “journal device.” The basic idea is to transform write requests to the produced geom into sequential writes Ivan Voras to the journal device. The class implements two kernel threads: A main worker thread to which I/O requests are delegated, and a helper thread used to asynchronously commit data from the journal to the data device. In regular mode, the journal device is divided into two areas, one of which is used to record data until it’s filled — at which point, it’s scheduled for asynchronous commit. A timed callout is scheduled that periodically triggers the swap/commit process. Two journal formats are implemented — one optimized for speed that emphasizes sequentiality of writes to the journal device, and another that conserves space by keeping metadata for the journal in one place. Unfortunately, the most used FreeBSD filesystem — the UFS — cannot be used with gjournal because this layer doesn’t distinguish metadata (for example, information about deleted but still referenced files) and requires a fsck run to correct references. The COW facility is functional and can be used for experimentation with filesystems.
Dr. Dobb’s Journal, December 2005
DDJ http://www.ddj.com
Wide-Character Format String Vulnerabilities Strategies for handling format string weaknesses ROBERT C. SEACORD
T
he ISO/IEC C Language Specification (commonly referred to as “C99”) defines formatted output functions that operate on wide-character strings, as well as those functions that operate on multibyte- character strings. The widecharacter formatted output functions include: fwprintf( ), wprintf( ), swprintf( ), vfwprintf( ), vswprintf( ), and vwprintf( ). (There is no need for snwprintf( ) or vsnwprintf( ) functions because the swprintf( ) and vswprintf( ) include an output length argument.) These functions correspond functionally to the multibyte-character formatted output functions (that is, the similarly named functions with the “w” removed) except that they work on wide-character strings such as Unicode and not on multibyte- character strings such as ASCII. (A multibyte character is defined by the ISO/IEC 9899:1999 as a sequence of one or more bytes representing a member of the extended character set of either the source or the execution environment. ASCII strings are represented as multibyte-character strings, although all ASCII characters are represented as a single byte.) Formatted output functions are susceptible to a class of vulnerabilities known as “format string” vulnerabilities. Format string vulnerabilities can occur when a format string (or a portion of a format string) is supplied by a user or other untrusted source. Listing One, for example, is a common programming idiom, particularly for UNIX command-line proRobert is a senior vulnerability analyst for CERT/CC and author of Secure Coding in C and C++ (Addison-Wesley, 2005). He can be reached at
[email protected]. http://www.ddj.com
grams. The program prints usage information for the command. However, because the executable may be renamed, the actual name of the program entered by users and specified in argv[0] is printed instead of a hardcoded name. By calling this program using execl( ), attackers can specify an arbitrary string as the name for arg[0], as in Listing Two. In this case, the specified string is likely to cause the program to crash, as the printf( ) function on line 6 of Listing One attempts to read many more arguments off the stack than are actually available. However, this could be much, much worse. In addition to crashing a program (and possibly causing a denial-of-service attack), attackers can also exploit this vulnerability to view arbitrary memory or execute arbitrary code with the permissions of the vulnerable program. For example, attackers can execute arbitrary code by providing a format string of the form: address advance-argptr %widthu%n
The address field contains a Little-endian encoded string; for example,\xdc\xf5\x42\ x01. The advance-argptr string consists of a series of format specifiers designed to advance the internal argument pointer within the formatted output function until it points to the address at the start of the format string. The %n conversion specifier at the end of the string writes out the number of characters output by the formatted output function. The %widthu conversion specifier advances the count to the required value. When processed by the format output function, this string writes an attacker-provided value (typically the address of some shellcode) to an attacker-specified address such as the return address on the stack. When the vulnerable function returns, control is transferred to the shellcode instead of the calling function, resulting in execution of arbitrary code with the permissions of the vulnerable program. A detailed description of format string vulnerabilities and possible exploits with multibyte-character strings is presented in my book Secure Coding in C and C++ (Addison-Wesley, 2005; ISBN 0321335724) and by Scut/Team Teso (see “Exploiting Dr. Dobb’s Journal, December 2005
Format String Vulnerabilities,” http://www .mindsec.com/files/formatstring-1.2.pdf). In this article, I focus on the vulnerabilities resulting from the incorrect use of wide-character formatted output functions. Environment Formatted output functions that operate on wide-character strings are also susceptible to format string vulnerabilities. To understand the effect of wide characters
“Formatted output functions that operate on wide-character strings are also susceptible to vulnerabilities” on format string vulnerabilities, you must understand the interactions between the program and environment. Here, I examine the mechanisms used to manage these interactions for Windows and Visual C++. Visual C++ defines a wide-character version of the main( ) function called wmain( ) that adheres to the Unicode programming model. Formal parameters to wmain( ) are declared in a similar manner to main( ): int wmain( int argc[ , wchar_t *argv[ ] [, wchar_t *envp[ ] ] ] );
The argv and envp parameters to wmain( ) are of type wchar_t *. For programs declared using the wmain( ) function, Windows creates a wide-character environment at program startup that includes wide-character argument strings and optionally, a wide-character environment pointer to the program. When a program is declared using main( ), a multibyte-character environment is created by the operating system at program startup. 63
Typically, a programmer that uses wide characters internally will use wmain( ) to generate a wide-character environment, while a program that uses multibyte characters internally uses main( ) to generate a multibyte-character environment. When a program specified for a multibytecharacter environment calls a widecharacter function that interacts with the environment (for example, the _wgetenv( ) or _wputenv() functions), a wide-character copy of the environment is created from the multibyte environment. Similarly, for a program declared using the wmain( ) function, a multibyte-character environment is created on the first call to _ putenv( ) or getenv( ). When an ASCII environment is converted to Unicode, alternate bytes in the resulting Unicode string are null. This is a result of the standard Unicode representation of ASCII characters. For example, the ASCII representation for a is \x60 while the Unicode representation is \x6000. Alternating null bytes creates an interesting obstacle for shellcode writers. Wide-Character Format String Vulnerabilities Listing Three illustrates how the widecharacter formatted output function wprintf( ) can be exploited by an attacker. This example was developed using Visual C++ .NET with Unicode defined and tested on a Windows 2000 Professional platform with an Intel Pentium 4 processor. Because the program was declared (on line 6) using wmain( ), the vulnerable program accepts wide-character strings directly from the environment that have not been explicitly or implicitly converted. The shellcode is declared on line 7 as a series of nop instructions instead of actual malicious code. Unfortunately, examples of malicious shellcode are not hard to locate on the Internet and elsewhere. The wchar_t array format_str is declared as an automatic (stack) variable on line 9. I’ll return to the mysterious float variable shortly. The idea of the sample exploit is to create a wide-character format string such that the execution of this string by the wprintf( ) function call on line 31 results in the execution of the shellcode. This is typically accomplished by overwriting a return address on the stack with the address of the shellcode. However, any indirect address can also be used for this purpose. The modulo divisions are to ensure that each subsequent write can output a larger value while the remainder modulo 0x1000 preserves the required low-order byte. Although most of the conversion specifications interpreted by the formatted output function are used to format output, 64
there is one that writes to memory. The %n conversion specifier writes the number of characters successfully output by the formatted output function (an integer value) to an address passed as an argument to the function. By providing the address of the return address, attackers can trick the formatted output function into overwriting the return address on the stack. The number of characters output by the formatted output function can be influenced using the width and precision fields of a conversion specifier. The width field is controlled to output the exact number of characters required to write the address of the shellcode. Because there are some practical limitations to the size of the width and precision fields, the exploit writes out the first word of the address (line 20) followed by the second word (line 27). The first word is written to 0x0012f1e0 and the second word is offset by two bytes at 0x0012f1e2. These addresses are specified as part of the format string on lines 29–30. The next trick is to get the argument pointer within the formatted output function to point to this address. Formatted output functions are variadic functions, typically implemented using ANSI C stdargs. These functions have no real way of knowing how many arguments have been passed, so they will continue to consume arguments as long as there are additional conversion specifiers in the format string. Once the formatted output function has consumed all the actual arguments, the function’s argument pointer starts to traverse through the local stack variables and up through the stack. This makes it possible for attackers to insert the address of the return address in a local stack variable or, as in this case, as part of the format string that is also located on the stack. The address is typically added at the start of the format string, which is then output by the formatted output like any other character. The widecharacter formatted output functions, however, are more likely to exit when an invalid Unicode character is detected. As a result, an address included at the beginning of a malicious format string is likely to cause the function to exit without accomplishing the attacker’s goal of executing the shellcode because the address is unlikely to map to valid Unicode. This is not a significant obstacle to determined attackers, however, because the address can be moved to the end of the format string as in lines 29–30 in a wide-character exploit. These addresses may still cause the function to exit, but not before the return address has been overwritten. The greatest problem introduced by moving the address pairs to the end of the string is that attackers must now Dr. Dobb’s Journal, December 2005
progress the argument pointer past the conversion specifiers used to advance the argument pointer to the start of the dummy integer/address pairs. This creates a race in that adding conversion specifiers increases the distance the argument pointer must be advanced to reach the dummy integer/address pairs. In ASCII, the conversion specifier %x requires two bytes to represent and, when processed, advances the argument pointer by four bytes. This means that each conversion specifier of this form narrows the gap between the argument pointer and the start of the dummy integer/address pairs by two bytes (that is, the four bytes the argument pointer is advanced minus the two bytes the start of the dummy integer/address pairs is advanced). The widecharacter representation of the %x conversion specifier requires four bytes to represent but only advances the argument pointer by four bytes (the length of an integer). As a result, the %x conversion specifier can no longer be used to gain ground on the dummy integer/address pairs. Is there a conversion specifier that can be used to gain ground on the dummy integer/address pairs? One possibility is the use of a length modifier to indicate that the conversion specifier applies to a long long int or unsigned long long int. Because these data types are represented in eight bytes and not four, each conversion specifier advances the argument pointer by eight bytes. Visual C++ does not support the C99 ll length modifier but instead provides the I64 length modifier. A conversion specifier using the I64 length modifier takes the form %I64x. This conversion specifier requires five wide characters or 10 bytes to represent but, as already noted, only advances the argument pointer by eight bytes. Now you are actually losing ground! Using a compiler that supports the standard ll length modifier (such as GCC) is not much better because the conversion specifier requires four characters or eight bytes. Another possibility is using the a, A, e, E, f, F, g, or G conversion specifiers to output a 64-bit floating-point number and thereby incrementing the argument pointer by eight bytes. For example, the conversion specifier %f requires two wide characters or four bytes to represent but advances the argument pointer by eight bytes, which lets attackers gain four bytes on the address for each conversion specifier processed by the formatted output function. Lines 11–14 in the wide-character exploit show how the %f conversion specifier can be used to advance the argument pointer. Line 11 also adds a single wide character (a) to properly align the argument pointer to the start of the http://www.ddj.com
dummy integer/address pairs. The only problem with the %f conversion specifier is that it can cause the abnormal termination of the program if the floating-point subsystem is not loaded — hence, the extremely unlikely declaration of a float on line 10. In theory, this problem could limit the number of programs that could be attacked using this exploit. In practice, most nontrivial programs load the floating-point subsystem. Venetian Shellcode Again, when a Windows program is declared using main( ), an ASCII environment is created by the operating system for the program. If a wide-character representation of an environmental variable is required, it is generated on demand. Because the Unicode string is converted from ASCII, every other byte will be zero. For example, if the ASCII string “AAAA”
is converted to Unicode, the result (in hexadecimal) is 00 41 00 41 00 41 00 41. This creates an interesting obstacle for exploit writers. Chris Anley has done some work (see “Creating Arbitrary Shellcode In Unicode Expanded Strings: The “Venetian” Exploit; http://www.ngssoftware.com/papers/ unicodebo.pdf) in creating Venetian shellcode with alternating zero bytes (analogous to Venetian blinds). While creating these programs by hand is quite troublesome, Dave Aitel’s makeunicode2.py and Phenoellit’s “vense” generator are both capable of automatically generating Venetian shellcode. Conclusion Wide-character formatted output functions are susceptible to format string and buffer overflow vulnerabilities in a similar manner to multibyte-character formatted out-
Listing One
28.
put functions, even in the extraordinary case where Unicode strings are converted from ASCII. Unicode actually has characteristics that make it easier to exploit functions that use these strings. For example, multibytecharacter functions recognize a null byte as the end of a string, making it impossible to embed a null byte (\x00) in the middle of a string. The null character in Unicode, however, is represented by \x0000. Because Unicode characters can contain null bytes, it is easier to inject a broader range of addresses into a Unicode string. There are a number of mitigation strategies for format string vulnerabilities. The simplest solution that works for both multibyte- and wide-character strings is to never allow (potentially malicious) users to control the contents of the format string. DDJ
wcscat(format_str, convert_spec);
// two dummy int/address pairs 29. wcscat(format_str, L"ab\xf1e0\x0012"); 30. wcscat(format_str, L"ab\xf1e2\x0012");
1. #include <stdio.h> 2. #include <string.h> 3. void usage(char *pname) { 4. char usageStr[1024]; 5. snprintf(usageStr, 1024, "Usage: %s
\n", pname); 6. printf(usageStr); 7. }
31.
wprintf(format_str);
32. return 0; 33. }
8. int main(int argc, char * argv[]) { 9. if (argc < 2) { 10. usage(argv[0]); 11. exit(-1); 12. } 13. }
DDJ
Listing Two 1. #include 2. #include <errno.h> 3. int main(void) { 4. execl("usage", "%s%s%s%s%s%s%s%s%s%s", NULL); 5. return(-1); 6. }
Listing Three 1. #include <stdio.h> 2. #include <string.h> 3. static unsigned int already_written, width_field; 4. static unsigned int write_word; 5. static wchar_t convert_spec[256]; 6. int wmain(int argc, wchar_t *argv[], wchar_t *envp[]) { 7. 8. 9. 10. 11. 12. 13. 14. 15.
unsigned char exploit_code[1024] = "\x90\x90\x90\x90\x90"; int i; wchar_t format_str[1024]; float x = 5.3; // advance argument pointer 63 x 4 bytes wcscpy(format_str, L"a%f"); // 2 bytes filler for (i=0; i < 63; i++) { wcscat(format_str, L"%f"); } already_written = 0x084d;
// first word 16. write_word = 0xfad8; 17. already_written %= 0x10000; 18. 19. 20. 21. 22. 23. 24. 25. 26. 27.
width_field = (write_word-already_written) % 0x10000; if (width_field < 10) width_field += 0x10000; swprintf(convert_spec, L"%%%du%%n", width_field); wcscat(format_str, convert_spec); // last word already_written += width_field; write_word = 0x0012; already_written %= 0x10000; width_field = (write_word-already_written) % 0x10000; if (width_field < 10) width_field += 0x10000; swprintf(convert_spec, L"%%%du%%n", width_field);
http://www.ddj.com
Dr. Dobb’s Journal, December 2005
65
Amazon Web Services Small devices meet large databases ASHISH MUNI AND JUSTIN HANSEN
S
canZoom is an application that lets you use mobile camera phones to launch services and applications simply by taking a photo of a barcode. For example, say you’re shopping at Fry’s Electronics and want to compare iPod prices with other retailers. You could take a picture of the barcode on the iPod packaging (or enter the barcode value manually) and ScanZoom finds out and reports on what that same iPod costs at competing retailers. Developed at Scanbuy (where we work), ScanZoom can be installed by first downloading the application onto a PC from http://www.scanzoom.com/, and then installing it on mobile phones (via BlueTooth, infrared, WAP, or SMS), or by embeddable microchips. In developing ScanZoom, we had two requirements that the data source needed to provide: • A large, extensive database. • Access to the Web without the need for monotonous manual entry or navigation. Because of the size of Amazon’s product database and its vast amount of information, Amazon’s freely available web service API was the first resource we turned to. In particular, Amazon’s ECommerce Service (ECS) provides access to all of the content on the product pages of the Amazon.com site in an XML format.
Ashish is the chief technology officer and Justin is a software engineer at Scanbuy. They can be reached at ashish@ scanbuy.com and [email protected], respectively. 66
From our perspective, Amazon’s ECS (http://www.amazon.com/aws/) revolutionizes what a web service can do for developers. The service gives you direct, real-time access to all of Amazon’s features, including consumer reviews, new and used product listings from individuals and companies, and a list of similar products that can be purchased along with almost all category-specific details that you can imagine. This service let us take ScanZoom to the next level and link mobile phones with the power of Amazon. Providing information that would not be readily available to users holding products in their hands was essential. If all we offered were those basic details, what good would the application be to the consumer? Users would be able to see most, if not all, of the basic information while scanning the barcode on the package. ECS provided us with extra information, which would be helpful to a consumer in making a decision while shopping. For example, many people read consumer reviews before selecting a product to purchase. Through ECS, we can offer shoppers quick access to this information so that they can read each and every review that Amazon has on a product. Shoppers can then make an informed decision without having to research the product before leaving home. The application is also able to show similar products that are offered so that consumers are able to see other options and read reviews on those items before making a decision. Still, pricing is one of the main reasons that consumers shop the Internet. However, comparing prices requires research, and many purchases are impulse buys made while shopping for other items and no research has been done. Through our application, users can instantly gain access to not only the price that Amazon offers, but to the price that retailers in the Amazon network provides. Because many of these retailers (Target and Circuit City, for instance) have large retail stores, they can go to another retailer to make their purchase at a discount. Users can also save more money by buying used items through Amazon. The used item prices are shown on the handset, along with the Dr. Dobb’s Journal, December 2005
seller’s rating and a description of item quality and condition. Specific product information is joined by the item’s packaging information, such as DVD movie run time, CD release date and track listing, and the like. Having all of this information combined into one data feed made it possible for us to pull up all of this information with a single search. The process of integrating the web service into our current web application was also fairly simple because Amazon offers
“You can further reduce the time it takes to return data to the mobile handset by searching in exact categories” a WSDL that we are able to connect to inside our C#.NET development environment. Creating a reference in the project to the URL of the ECS WSDL is all it took to have access to all the data types and functions necessary to gather our product information. Example 1 is a sample connection and query that returns an Amazon Item data type containing all the products from the search, along with the relevant item information. That’s all that’s involved in fetching Amazon’s product information. This query is also very customizable. Because we are dealing with mobile devices with limited memory and bandwidth, we need to make sure that the page being retrieved by the mobile browser is as compact as possible. We are able to efficiently do that by limiting the searchRequest.ResponseGroup to an array of only the response groups that are necessary for what is being displayed on the phone. For example, if you only need to know the item’s price from Amazon along with some basic item details (such as category), you can simply have the response group set http://www.ddj.com
1 2 3 4 5 6
string subID = ""; AWSECommerceService service = new AWSECommerceService(); ItemSearch itemSearch = new ItemSearch(); ItemSearchRequest searchRequest = new ItemSearchRequest(); searchRequest.Keywords = keywords; string[] responseGroup = newstring[]{"OfferFull","ItemAttributes","Reviews","Images"}; searchRequest.ResponseGroup = responseGroup; searchRequest.SearchIndex = "Blended";
7 8 9 10 ItemSearchRequest[] searchRequests = newItemSearchRequest[]{searchRequest}; 11 itemSearch.SubscriptionId = subID; 12 itemSearch.Request = searchRequests; 13 ItemSearchResponse response = service.ItemSearch(itemSearch); 14 Items info = response.Items[0]; 15 Item[] items = info.Item; 16 return items[0];
Example 1: Sample connection and query that returns an Amazon Item data type.
REST versus SOAP
T
he proliferation of web services has brought out an interesting focus on REST versus SOAP. Those in favor of REST argue for HTTP’s built-in security (SSL). On the other hand, SOAP offers flexibility in delivery through other modes of transport by storing address information in the SOAP envelope. While Amazon ECS offers both scenarios to developers, we opted for SOAP because it provides consistency across different applications. This feature was especially important in developing ScanZoom since the screen of the cell phone is limited in both size and resolution compared to that of typical displays. Because of this constraint, we couldn’t simply redirect users to an HTML web page (such as Amazon’s
to “Small” (also offered are “Medium” and “Large”). Those response groups contain an array of smaller response groups. Instead of picking “Medium” or “Large” when you need more than just the basic information, you call on only those response groups that are needed, such as the ones that are listed on line 6 in Example 1. You can further reduce the time it takes to return data to the mobile handset by searching in exact categories with more exact search criteria. In Example 1 (line 8), we are using a Blended SearchIndex. However, you could specify Books or Movies instead. We don’t use this query when doing the initial search based on UPC or ISBN, but when we are requesting the “Reviews” or “Similar Products,” we use it to minimize the irrelevant results and make the data set smaller. Most of the development we did to integrate ECS into our web application was as simple as Example 1. However, we did run into a snag here and there. For exhttp://www.ddj.com
product pages) because navigating it would be nearly impossible. In addition, many of the stock web browsers available on mobile devices do not support the full HTML collection and require a special XHTML Mobile Profile compliant page. Because Amazon provides access to its information through XML, we were free to format its content in a way that made it simple for the end user to understand and navigate, even on primitive cell-phone browsers, while still allowing the functionality that one would expect with more advanced PDA-type handsets. ECS also let us handle browsercompatibility issues, increasing the number of handsets that we can support. —A.M. and J.H. ample, we needed to display products from different categories on the same page template. The item information that is returned in the Item data type from ECS has different names for item details depending on the item category. For instance, the artist of a CD would be reached from Item.ItemAttributes.Artist, while the writer of a book would be Item.ItemAttributes.Writer. To solve this issue, we find the Item.ItemAttributes.ProductCategory, then list the item details based on that category. This was the only issue that we faced and it was definitely not a major issue, just a small bump in the road. Using ECS, our application offers users a mobile shopping and pricing guide that is simple and fast to use. Developing the solution was almost as easy and let us offer users something that they couldn’t previously get on their cell phone without a lot of keyboard pecking and squinting at a small screen. DDJ Dr. Dobb’s Journal, December 2005
67
WINDOWS/.NET DEVELOPER
Enterprise Application Logging Using the SQL Server 2005 Service Broker JIM MANGIONE
W
hile we routinely standardize project details such as naming conventions and abstraction layers of corporate architectures, standardized logging often gets overlooked. Consequently, each application ends up logging messages differently. Can we do better? Absolutely. For the enterprise to accurately gauge the health of its IT assets across applications and gather pertinent metrics to support those findings, we need true enterprise-level logging. In this article, I propose a common way to meet this objective, using Microsoft’s SQL Server 2005 as an example. In the process, I present a “Hello Broker” WinForm client using the Log4NET framework. The complete source code for the client-testing application (available electronically, see “Resource Center,” page 4) posts test messages to each of the log severities so you can trace them through the queues. The requirements for a logging system include: • A common schema for log messages. • A common set of severities. Jim is a senior consultant in Global Medical Applications at Wyeth Pharmaceuticals. He can be reached at mangionej @yahoo.com. 68
• A centralized, loosely coupled logging service with a single point of entry and published API. • A scalable architecture. • The ability to dynamically set up and change application severities/destinations centrally (without touching each individual application configuration). These specifications— especially the scalability and asynchronous features — point to a traditional middleware, queuing-type implementation. Logging Service Architecture Figure 1 is a high-level view of the proposed logging service. Again, I am implementing a queuing-based system, which aids in scalability and performance of client applications using this service. At the core is the Central Log Queue, which accepts all log message types from all registered applications. Messages are published to this queue from the public API Log Publisher. This is the only way a message can enter the queue. Other publishers may exist to translate proprietary log messages to the standard message format our service accepts, but they must then call the Log Publisher to post the messages. (This is demonstrated in the Log4NET client implementation.) The Central Log Queue has a single subscriber type, a message router that consumes each message and interrogates its contents to determine what destination queue(s) to route to. Messages are routed based on their severity and the application’s configuration details. Each application that wants to use this service must register by adding information into the log service metadatabase. For routing purDr. Dobb’s Journal, December 2005
poses, a mapping of each severity to one or more destinations is all that is necessary. Logs that are published with severities that aren’t mapped simply vaporize — they are consumed and ignored so the queue doesn’t jam up.
“SQL Server 2005 includes a reliable messaging framework within its database engine”
Each destination is also a queue. This again facilitates a scalable architecture where the service can distribute the workload of more popular destinations across multiple queues. Each destination queue has its own subscriber type that, upon consuming a message, performs the particular function that places that message at the end point — the real destination. Here, the severity can be ignored for purposes of workflow because these subscribers are only interested in dumping all messages to its relevant destination. I have defined three destinations: http://www.ddj.com
The Service Broker SQL Server 2005 includes a reliable messaging framework within its database engine called “Service Broker.” It has most of the common queuing functionality you would expect from messaging middleware, including support for transactions, message ordering, and the ability to scale and provide service programs, which consume messages and run multiple instances of themselves based on the load on the queue. Comparisons can be made between this and the Java Messaging Service, with those service programs acting like MessageDriven Beans (MDBs). When opening the SQL Server Management Studio, you notice some sections in each user database; namely, a section called “Service Broker” that has some of these object types: • Message Type. Defines the name and type of a particular message that is used by a service. • Contract. Defines which message types to use and what the associated end points are between two services. • Service. Defines a set of related tasks based on a particular queue and contract, and links a conversation to a queue. • Queue. The persistent storage for messages. An internal table type. Another important part of the broker is how it communicates. “Conversations” are the means by which messages are sent to a queue (by way of a service) and how responses are sent back to the originator http://www.ddj.com
Publishing to the Central Log Queue Assuming the database is already created, the first main Service Broker component to build is the Queue. Before executing a one-line SQL statement that does this,
however, you need to define what the XML log message that gets queued looks like; see Example 1. Application_Name and Severity_Cd must exist in their respective tables, and the application must have a destination configured for this particular severity. Now you can setup the appropriate plumbing to move this log message into a queue (including the queue object itself); see Example 2. The WELL_FORMED_XML message type validation ensures that nonXML
Log4NET Log Publisher
ADONETAppender
Log Publisher Log4NET Framework
Publish Message Main Queue
Figure 2 defines the metadatabase used for configuring individual applications to run in the log service. This is a straightforward design in which the schema consists of the core entities for configuring each application. The Application table stores information about each registered application, including e-mail information for that type of destination, and current logging levels. Severity defines the five supported levels, from INFO to FATAL. Destination defines each supported destination type and binds that to a Service Broker service. Putting it all together is the Central_Log_Configuration table that defines what each application’s severity-todestination mappings are.
(in the case of dialogues). For this logging system, you don’t need to worry about true dialogues, only about sending/receiving messages. Although this still must be done through conversations, the implementation is simplified.
Central Log Queue
Consume Message
Message Router
Configuration Database
App Config
Route Message To Queues Destinations
• E-mail, where recipients are defined in the application configuration of the service. • Log database, a central repository to capture important severities from all applications for reporting and analytics. • Windows application events, to act as a gateway to other monitoring or auditing tools if applicable.
E-mail Queue
Log DB Queue
Consume Message E-mail Handler
Windows Event Queue
Consume Message Log DB Handler
Windows Event Handler
Insert Message To Central Log Repository
Send Message To Recipients
Consume Message
Post Message To Windows Event Log Event msg Event msg Event msg
Log Database
Figure 1: Proposed logging service. Central_Log Central_Log_Id P
Application Application_Id Application_Name Application_Desc Logging_Level Email_Recipients Email_Admin_URL
Message_Date Message Application_Id Severity_Id
P
Central_Log_Configuration Application_Id Severity_Id Destination_Id Config_Desc
Severity Severity_Id Severity_Cd Severity_Desc
Destination Destination_Id Destination_Name Service_Name
Figure 2: The metadatabase. Dr. Dobb’s Journal, December 2005
69
code doesn’t get into the queue. There are also other choices, including one that defines a particular schema to validate against. Creating a contract for the system is simplified because this is a one-way message only — no response is needed. Instead of SENT BY ANY, you specify a message type for sending by INITIATOR, and another message type for sending to TARGET. Finally, the creation of the service binds everything together. How do you get a message published? This is done by the Log Publisher stored procedure in Listing One. Sending an XML message isn’t that much work. First, a dialog started against the LogService defines which queue the message goes into and which contract is used for the conversation. You still need to define a from/to service, but because this isn’t a true dialog, they can be the same. A unique message handle is received from this statement. That handle is then used to send the message. Log messages sit in the queue until they are consumed by a service program, which attaches itself to the queue and receives the messages for processing. In this case, the processing is simply publishing to another queue. Routing Messages To Destination Queues Once messages are in the Central Log Queue, they are received by a program that — based on the application and severity— routes that message down to a destination queue. To do this, you’ll notice new Service Broker syntax in SQL. A basic SQL statement to receive (remove) a message off a queue looks like this: RECEIVE cast(message_body as nvarchar(max)) FROM [CentralLogQueue]
If you want to block on receive until a message arrives, then you can wrap the aforementioned code in a WAITFOR statement and define a time interval to fall through: WAITFOR ( RECEIVE cast(message_body as nvarchar(max)) FROM [CentralLogQueue] ), TIMEOUT 5000
Armed with these basic statements to retrieve messages, you can now build the most complex part of the system — the Router stored procedure in Listing Two. The core algorithm is: 1. Get a message off the queue. 2. Look up the application’s configuration and determine what destinations to publish the message to based on the severity of the message. 3. For each destination (service), create a dialog and send the message. Once the message arrives, OpenXML( ) returns the application and severity. From this, you can query the configuration tables to return the destination and service names you need to publish to. On each destination, you dynamically bind the service name to the begin dialog, and using the same LogServiceContract used for the original Publisher procedure, you send the exact same LogRequest message you received to the destination queues defined by the service name. Note the use of LogServiceGUID. At the beginning of this procedure, you cache this value. If you were hosting the same service (same name) on multiple databases, you would need to choose which GUID to use. Once you have the router that will keep listening for messages, it’s invocation time. You can run the thing via command line
<Application_Name>APP_TEST <Severity_Cd>INFO <MessageDateTime>1/1/2005 12:12:12 <Message>This is an INFORMATIONAL TEST message!
Example 1: Defining an XML log message. CREATE MESSAGE TYPE [LogRequest] VALIDATION = WELL_FORMED_XML CREATE CONTRACT [LogServiceContract] AUTHORIZATION [dbo] ([LogRequest] SENT BY ANY) CREATE QUEUE [dbo].[CentralLogQueue] CREATE SERVICE [AppEventLogService] AUTHORIZATION [dbo] ON QUEUE [AppEventDestinationQueue] ([LogServiceContract])
Example 2: Moving a log message into a queue. 70
Dr. Dobb’s Journal, December 2005
to test it, but there’s a better way to do this in production — let the queue manage it: ALTER QUEUE [CentralLogQueue] WITH ACTIVATION ( STATUS = ON, PROCEDURE_NAME = [SPRouter], MAX_QUEUE_READERS = 10 , EXECUTE AS N'dbo' )
The queue will automatically activate SPRouter once a message is published. As the load on the queue increases, more instances of the service program are invoked up to the max readers (in this case, 10). Once these programs are running, however, they cannot be shut off unless the queue is disabled. If the load on the queue decreases, altering the activation status to OFF keeps new instances of the program from running (if max isn’t hit), but will not shut down live ones. This functionality exists to keep the broker from killing processes in midstream (because it probably wouldn’t know if the program is waiting for a message or churning through one). If the broker’s event model was easily exposed and delegation was used to invoke the service program, you would not need to continually return to the waitfor. This would let the engine adjust the instances downward without concern over where the process is. Destination Service Programs Using the main algorithm from the Router, you can cut out the middle part (sending to another queue), replacing it with the specific code to send e-mail, insert into the Central Log Database, and post to the window’s event log. Listings Three, Four, and Five (all available electronically) accomplish these tasks, respectively, although you may have your own methods for e-mail and event logging. Testing the Service Using Log4NET Apache’s Log4NET (http://logging.apache .org/log4net/) is an open-source logging framework based on the Log4j project. The main goal of the framework is to allow easy inclusion of logging into applications, with the ability to define many destinations for outputting the logs, and filtering them based on severity. Within five minutes, you can introduce minimally a console-based logging mechanism into your .NET application. With a little more effort, you can extend that to such destinations as rolling files, SMTP, remoting, Window’s event logs, and databases, which is where the integration point with our Central Logging System occurs. Log4NET uses appenders to define destination paths to send log messages to. One such appender is ADONetAppender, which allows the writing of logs http://www.ddj.com
to a database either through defined SQL or stored procedures. Say you want any severities of Warning, Error, or Fatal to be sent to our Enterprise Logging Service (and keep the verbose logging of INFO and DEBUG on the client side only). What is needed is to define in the application’s configuration file an instance of the ADONetAppender that maps to the Log Publisher stored procedure. This creates that bridge into the Central Log Queue. Because the Log Publisher expects an XML Log Message, you need a translation
of the Log4NET’s log class prior to calling the procedure. To ensure all the messages are translated the same from any number of Log4NET implementations, you do this in a Log4NETLogPublisher stored procedure; see Listing Six (available electronically). The ADONetAppender configuration that defines the necessary plumbing needed to pass the log fields into the stored procedure is defined below that. The @ApplicationName is passed in here, as well as the level being set to WARN. In the Central Logging configuration, you need to have at least one destination
Listing One -- This procedure will receive an XML log record and submit to the Central -- Queue via the Log Service.
DDJ
-- setup an internal representation of this XML doc exec sp_xml_preparedocument @xml_doc_handle output, @message_body -- get the destination names based on application and severity of this log -- and load into temp destination table insert into @xml_log_app_and_severity select application_name, severity_cd from OpenXML(@xml_doc_handle, '/ROOT/CentralLogMessage',2) with (Application_Name varchar(64), Severity_Cd varchar(32))
begin transaction begin dialog conversation @msgHandle from service [LogService] to service 'LogService' on contract [LogServiceContract]; send on conversation @msgHandle message type [LogRequest] (@msgXML) end conversation @msgHandle commit
-- lookup the destination(s) via the configuration insert into @destinations select d.destination_name, d.service_name from destination d, central_log_configuration clc, application a, severity s, @xml_log_app_and_severity x where x.application_name = a.application_name and x.severity_cd = s.severity_cd and clc.application_id = a.application_id and clc.severity_id = s.severity_id and clc.destination_id = d.destination_id and a.logging_level >= s.severity_id
Listing Two CREATE PROCEDURE [dbo].[SPRouter] WITH EXECUTE AS CALLER AS -- This is a Service Program that will receive Log Messages as xml from the -- Central Queue -- (LogService) and route them depending on the Log_Configuration metadata to -- the appropriate -- Destination Queues (App, DB and/or Email Services). --- Workflow: -- a) receive an xml log message from the CentralLogQueue -- b) for each destination_id that is setup in the Log Configuration for the -Application_Name and Severity_Cd tags in the message log, post the -xml log message to that destination queue. -declare @conversation_handle uniqueidentifier declare @message_body nvarchar(MAX) declare @message_type_name sysname
-- publish the log to each destination by iterating through the table declare cDestinations cursor fast_forward for select * from @destinations open cDestinations fetch next from cDestinations into @destination, @service while @@fetch_status = 0 begin -- publish message to designated service -- @destination name is only needed for debugging purposes begin transaction begin dialog conversation @msgHandle from service @service to service @service, @LogServiceGUID on contract [LogServiceContract]; send on conversation @msgHandle message type [LogRequest] (@message_body) end conversation @msgHandle commit transaction
declare @xml_doc_handle int; -- used by XPath to create a doc in memory declare @xml_doc varchar(MAX); -- receives xml doc from the sp_preparedoc call -- database ID for these services (assume they all exist in same database) declare @LogServiceGUID uniqueidentifier; -- store the results of this xml log into a temp table for querying to find -- destinations declare @xml_log_app_and_severity table ( application_name varchar(32), severity_cd varchar(32) )
-- get the next row (service) to publish this message too fetch next from cDestinations into @destination, @service
-- will store each destination and service that this log is going to goto declare @destinations table ( destination_name varchar(32), service_name varchar(32) )
end close cDestinations deallocate cDestinations
for working on individual rows from the destinations table above @destination varchar(32) @service varchar(32) @msgHandle uniqueidentifier
-- cache this for use when publishing the messages to destination queues SELECT @LogServiceGUID = service_broker_guid FROM sys.databases WHERE database_id = DB_ID('LogService'); -- Keep this alive. Else the broker will call this procedure for every message. while ( 1=1 ) begin -- receive the next message off the queue waitfor ( receive top(1) @message_body=message_body, @message_type_name=message_type_name, @conversation_handle=conversation_handle from [CentralLogQueue] )
http://www.ddj.com
Conclusion For organizations looking to upgrade to SQL Server 2005 and do more in respect to enterprise-application logging and exception handling, this project could act as a conduit for exploring the feature set of Microsoft’s new Service Broker to produce a trimmed-down but functional and scalable logging service.
-- insure the message is of the proper type (should ALWAYS be) if @message_type_name = 'LogRequest' begin
CREATE PROCEDURE [dbo].[SPLogPublisher] @msgXML [nvarchar](max) WITH EXECUTE AS CALLER AS declare @msgHandle uniqueidentifier
-- used declare declare declare
mapped to WARN, ERROR, and FATAL for this application.
-- clean up temp tables delete from @xml_log_app_and_severity delete from @destinations -- clean up exec sp_xml_removedocument @xml_doc_handle end else begin -- wrong type of message, or an error occurred. print N'Central Logging: Invalid message type received for DIALOG HANDLE: ' + cast( @conversation_handle as varchar(256) ) + N' XML MESSAGE: ' + cast( cast( @message_body as XML ) as varchar(MAX)); end end -- loop
Dr. Dobb’s Journal, December 2005
DDJ 71
EMBEDDED SYSTEMS
Memory Management & Embedded Databases
E
fragmentation strategies employed by general-purpose allocators impose CPU overhead that is often prohibitive for embedded systems, while custom memory managers can offer lightweight defragmentation more suited to embedded applications’ resource constraints and required short response times. In this article, we examine generalpurpose and custom memory-allocation strategies, illustrating their usage and advantages/disadvantages generally as well as their applicability for particular database management programming tasks. Many of the concepts derive from our experience creating the eXtremeDB in-memory database and its eXtremeSQL extension (http://www.mcobject.com/).
Andrei is principal architect of McObject. Konstantin is a software engineer with Borland. They can be reached at gor@ mcobject.com and konstantin.knizhnik@ borland.com, respectively.
List Allocators List allocators are perhaps the most widely used and well-known allocator algorithms, and often form the foundation for general-purpose memory managers that handle unpredictable allocation patterns. This type of algorithm organizes a pool of contiguous memory locations (often called “free holes”) into a singly linked list. The allocator services a request for memory allocation by traversing the list looking for a large enough hole. Several strategies are possible when searching for an area that satisfies the request for memory allocation of a given size. These include first-fit searches, in which the allocator walks a linked-list to find the first available memory hole; next-fit, which begins searching where a previous search left off, rather than from the beginning of the list; and quick-fit, in which the allocator uses its own list of common memory sizes to quickly allocate a block of memory that is large enough (but perhaps larger than needed). Almost all list allocator implementations suffer from a fragmentation problem. If an application intensively allocates and frees objects of different sizes and different lifetimes, then, in time, the list will only contain a large number of small holes. To battle fragmentation, list algorithms usually implement techniques that merge small holes together, or that sort the list by hole size, enabling the first fit algorithm to quickly locate a free hole that best matches the allocation request size. But efficient
Building an infrastructure with optimization in mind ANDREI GORINE AND KONSTANTIN KNIZHNIK mbedded databases in general, and in-memory databases in particular, are especially dependent on the quality of their memory-management algorithms. Designed for use in resourceconstrained embedded systems, the features, performance, and predictability of these databases depend heavily on the efficiency of algorithms for allocating precious memory. The performance cost of general-purpose allocators, such as the Windows C runtime or glibc allocator, is prohibitive for many embedded applications, and the memory overhead is often excessive. As a result, many embedded applications, including database management systems, implement custom memory managers for optimization. (In this article, we use the term “application” to refer to various programming tasks. The application, for example, can be a filesystem or operating-system kernel.) A single database system often utilizes numerous allocation algorithms to address specific internal tasks such as infrastructure for data layout, heap management, and SQL parsers and optimizers. Generally speaking, allocators keep track of which parts of memory are used and which are free. The design goal of any allocator is to minimize wasted memory space, balancing the amount of wasted space against the processing time required to recover it. A major target of allocators is to limit or mitigate the fragmentation that occurs when applications free memory blocks in any order. De-
72
Dr. Dobb’s Journal, December 2005
defragmentation techniques come at the price of extra per-object overhead. Defragmentation’s performance and per-object memory overhead must be balanced against any gains in efficient memory use. Often the more task-specific allocators we describe here can provide greater efficiency than generic list allocators.
“List allocators are perhaps the most widely used and well-known allocator algorithms” Block Allocators List-based allocators’ per-object overhead can be prohibitive in resource-constrained embedded environments. Many applications tend to allocate a large number of small objects of the same size. Typical examples of such allocations include small scalar values such as date/time, and abstract syntax tree nodes used by various parsers. Block allocators handle such objects very efficiently with minimal overhead, and eliminate fragmentation by design. The idea of a block allocator is straightforward. A block allocator is given a quantity of memory (we’ll call this the “large block”), divides it into equal-size pieces, and organizes them in a linked-list of free elements. To serve a request for memory, the allocator returns a pointer to one piece and removes it from the list. When there are no more elements in the “free list,” a new large block is selected from the memory pool using some other allocator (such as a list allocator). The new large block gets divided by the block allocator, elements are put into a new linked-list, and allocation requests are handled from this newly created list. When an object is freed, it is placed back into its original “free list.” Because all allocated objects in a given list are of the same size, there is no need for the block allocator to “remember” each element’s size, or to locate neighboring http://www.ddj.com
chunks to merge them. Listing One implements a simple block allocator. In other cases, more complex implementations of the block allocator algorithm are justified to improve memory-management efficiency. Often, application processing is divided into multiple stages. Objects allocated at each of these stages are not necessarily needed during subsequent stages. The block allocator used during each particular stage can be designed to return the unused blocks back to the memory pool. For example, a compiler includes three distinct processing phases: • Parsing, in which the compiler builds the abstract syntax tree (AST). • Analyzing language statements and mapping them to an internal representation. • Generating machine code. The block allocator is well suited to manage the compiler’s allocations, which contain a large number of similar objects, such as AST nodes (parsing results). With the simplest block allocator, however, the compiler would not be able to reuse the space allocated for the AST nodes, even though they are not needed during the subsequent stages. However, a slightly more sophisticated implementation would free the AST at the end of the first phase, allowing the compiler to reuse the memory in the second and third phases. The most basic block allocators satisfy allocation requests only for objects that fit into their predetermined element size, making such algorithms useful only when the allocation pattern is known in advance (for example, when the application always allocates 16-byte objects). In practice, many memory managers, including database memory managers, need to satisfy requests of several allocation patterns. To utilize the simplicity advantage of the block allocator algorithm while meeting the application’s need to allocate memory in variously sized chunks, a block allocator is often combined with some other technique into a hybrid memory manager. For example, the block allocator can maintain multiple lists of different-sized elements, choosing the list that is suited for a particular allocation request. Meanwhile, the blocks themselves and other large objects — those that ex-
ceed the chunk size of any of the blocks — are allocated using another generalpurpose allocator (for example, a page allocator or a list allocator). In such an implementation, the number of allocations (objects) processed by the block algorithm is typically orders of magnitude times higher than those made by the general-purpose malloc( ), resulting in startling performance improvements compared to using malloc( ) on its own. In such a hybrid memory manager, the large block size depends on the typical allocation request and can be chosen by the application. To better illustrate the algorithm’s functioning and demonstrate one practical approach, assume the block size is 512 bytes. Also assume the minimum size ever desired for allocation is 16 bytes and that objects are aligned on 8-byte boundaries. This alignment assumption is reasonable since on many hardware architectures the data alignment is an absolute requirement, and many others perform aligned data access much faster. Objects that are larger than one-half of the block size are allocated by a generalpurpose allocator. For objects smaller than one-half of the block size, the memory manager uses one of the block allocator chains. But how many blocks should the memory manager maintain, and what are the optimal sizes of the elements within these blocks? The maximum size of the object allocated by the block allocator is 256 and there can be exactly two such objects allocated from the block. Let’s call this chain a “256 chain.” To be able to allocate three objects out of a block, the element size would be 168 — let’s call it a “168 chain.” There is no sense in creating separate chains for 250 or 160 sizes, since they would require the same number of blocks as our 256 and 168 chains, respectively. If the process is continued further, it creates 13 chains with 512/4, 512/5,…512/16 chunk sizes (see Figure 1). The two tables in Example 1 describe the data layout in Figure 1. The first array specifies the block and the second specifies the sizes of the block elements. Note that since we assumed 8-byte alignment, it’s possible to divide the aligned object size by 8 and reduce the dimension of the arrays to 32 instead of 256.
const int storage::block_chain[32] = { 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9, 10, 10, 10, 10, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12 }; const int storage::block_size[32] = { 16, 16, 24, 32, 40, 48, 56, 64, 72, 80, 96, 96, 128, 128, 128, 128, 168, 168, 168, 168, 168, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256 };
Example 1: Description of the data layout in Figure 1. http://www.ddj.com
Block_0
Listing Two implements the allocate( ) and free( ) methods. Note that the allocator keeps the size of the object in a hidden header field within the object. This information allows the allocator to free the object by passing only the object pointer to the free( ) method. It is possible (although not necessarily practical) to avoid the extra object header overhead by requiring the blocks to be aligned on the block size (512 bytes in our example). It would also require reserving some space for each block within the block itself (block_header below) that would contain the block size. The size of the element would then be calculated like this: void free(object* obj){ size_t size = ((block_header*) ((size_t)obj & ~(block_size-1))->size; ... }
This would also reduce the maximum size of the object that could be allocated from a page — it would only be possible to keep two 248-byte objects. Stack-Based Allocators Memory allocators would be much simpler to design and exhibit higher performance if they only had to allocate objects and not free them. Given a block of memory, such an allocator would just advance a pointer, verifying there is enough space left in the block to satisfy an allocation request. Moreover, such an allocator would impose no memory overhead. The downside of this policy is obvious — if the memory manager is not guarding, the application could run out of memory. However, some embedded applications benefit from incorporating custom allocators that allocate objects but never release them. This approach is justified when the number of allocations and/or the total amount of required memory is known in advance and limited. A slight variation on this algorithm is particularly useful when memory requirements can be divided into a first phase in which objects are allocated, and a second in which they are deallocated. In this case, the stack pointer can simply be rewound, effectively deallocating the objects. This allocation pattern is typical for a SQL engine, which needs to free the Block 1
2x256 Chunks
3x168 Chunks
Block 13
32x16 Chunks
Figure 1: Data layout.
Dr. Dobb’s Journal, December 2005
73
memory allocated while parsing a SQL statement, but only deallocates once the statement is processed. A memory manager built on this twophase strategy can be highly efficient, yet allows reuse of the memory pool. It maintains a pointer to the current position in a memory block to allocate objects within the block. To allocate an object, the pointer is incremented by the requested size, and the old value of the pointer is returned to reference the allocated memory. When there is no more space available in the current block, a new block is allocated out of the application’s memory pool. This is done by some generalpurpose allocator such as the standard malloc/free. Blocks used in this process are kept in a linked-list. The deallocation part of the algorithm is also quite trivial to implement, and fast in execution: All blocks are simply released using the general-purpose deallocator (for instance, free( )). The simple stack allocator in Listing Three (available electronically; see “Resource Center,” page 4) implements the two-phase strategy previously described. It starts with a fixed amount (block) of memory. The allocator maintains and the application marks (remembers) the current stack position pointer that identifies a segment on the stack associated with the processing. When the application completes the processing and no longer needs the objects, it releases (frees) the stack segment, causing the allocator to reset the stack pointer back to the mark. While the current process is still active, the application may initiate another operation that also uses the allocator. When this starts, the allocator marks the current stack position that identifies a new segment for use. The stack segments never overlap: The application never allocates more objects from the first segment until the second segment is removed from the stack (see Figure 2). The deallocation of segments is done in LIFO order; the segments marked last must be deallocated first. The allocation/deallocation pattern just described may sound impractical for many applications, due to the rather narrow range of tasks that it can serve. However, in the context of a relational database engine, this pattern is typical. For example, when evaluating a condition, the temporary results of subexpression evaluations need only be kept until the condition evaluation is comSegment_1
plete. The stack can be marked before starting to evaluate a condition, and reset to the mark after results are returned. Stack allocators are superior when an application, as a natural byproduct of its operation, keeps track of the order in which objects are allocated. For some applications, such as those that perform recursive processing, a stack-based allocator simplifies design and improves performance. A database engine is a vivid example, since many of its algorithms use recursion. If the engine uses an allocator that requires explicit deallocation of individual objects, it must track each object’s lifespan, as well as references to the object by other objects. Failure to do this results in memory leaks and infamously “wild” dangling pointers. In contrast, setting a mark on the stack before processing a statement, and rewinding to the mark after processing, is as simple and fast as it gets. When the stack allocator is used, a SQL engine controls exactly how objects are allocated. Therefore, it is able to enforce the LIFO order of allocation/deallocation procedures. However, when it is necessary to use objects created inside the SQL engine outside its scope, it is not always possible to preserve the LIFO order. For example, in a SELECT statement, an application executes a query and starts iterating over the result set allocated by the engine via its stack allocator. While processing the result set, the application can execute another query, then close the first result set and start processing the second one. Two result sets overlap on the stack and the database is not capable of maintaining the LIFO order for allocation/deallocation procedures. To handle this scenario, the allocator algorithm can be extended by keeping an identifier (an integer number) of the stack segment that is currently being used (the second result set segment in our example). The allocator also maintains a count of the stack segments. When the application attempts to free one of the segments, the allocator compares the identifier of the current segment with the one being deallocated. If the identifiers are equal, the allocator resets the stack. Otherwise, the counter is decremented and the reset is postponed until the count is zero. In our example, the stack is only reset when both result sets are closed. Again, Listing Three illustrates the allocator.
Stack-based allocators are fast and impose very little overhead. For data management, the stack model can be used to allocate short-lived objects that can be released all at once — during SQL statement execution — for example, when all memory is released upon the statement commit or rollback. A useful byproduct of this approach is improved safety: With a stackbased allocator, it is impossible to accidentally introduce a memory leak through improper deallocation because the application need not track individual allocations. Thread-Local Allocators Memory allocation performance in multithreaded environments is a challenge for any application. In particular, multithreaded applications running on multiprocessor systems can bog down when performing many allocations, with lock contention in the default allocator causing the bottleneck. To synchronize access to its internals, the C runtime allocator (malloc( )/free( )) uses a mutex that is signaled every time the allocator is used. By itself, the mutex is not all that expensive, in performance terms. On most OSs, it is implemented via an atomic check-and-set instruction. However, resource conflicts arise when multiple threads running on different CPUs attempt to access the allocator concurrently: Each thread attempts to acquire the mutex, creating a lock conflict. To resolve the conflict, the OS has to do a context switch, suspend the thread that attempted to access the allocator, and insert it into the kernel’s waiting queue. When the allocator is released, the current thread is allowed to run and access the allocator. The large number of context switches degrades performance. (Without addressing the underlying issue, the same application would perform much better on a single CPU.) One way to resolve these conflicts is to create an allocator that simply avoids them. Most modern operating systems support the concept of per-thread storage, or a memory pool that is assigned to an individual thread. A thread allocator associates a memory block with each thread and services the thread’s memory requests from this block, without interfering with other threads’ allocation requests. When the thread allocator runs out of memory, the default allocator assigns it another block. Obviously, the number of lock conflicts is significantly decreased.
Segment_2 S Bytes Long 1
1
0
0
...
1
0 0 ... 0 0
1
QS zeros Mark_1
Mark_2
Mark_3
Figure 2: The stack segments never overlap. 74
Figure 3: The allocator searches its bitmap. Dr. Dobb’s Journal, December 2005
http://www.ddj.com
One scenario requiring special attention is threads sharing objects: An object allocated in one thread is later freed in another. The thread allocator handles this by detecting the attempt to free the object and redirecting this request to the thread that performed the original allocation. This could take the form of linking all objects allocated by one thread into a linked-list. This would require a mutex to synchronize access to the list, so this mechanism would be efficient only if the number of objects migrating between threads is relatively small compared to the number of stationary objects. It should be mentioned that even intentionally allocating objects in one thread and freeing them in another can result in sharing of cache line among processors, and its attendant performance degradation. In database systems, the “stay-at-home” behavior of objects, discussed earlier, is usually enforced by the DBMS itself. Each thread performs its database access in the context of a database transaction, and these are isolated from one another by the database transaction manager. Thus, in database design, it is often possible to entirely skip the issue of migrating objects in the allocator. A stack-based allocator (such as those described in this article) usually represents a good choice for implementing a thread-local allocator. The combination of the stack and the thread-local storage allows development of synchronization-free memory managers that avoid lock contention when accessing memory, resulting in improved database performance on multiprocessor systems. Bitmap Allocators A bitmap allocator acts on a memory pool by allocating objects in prespecified units (an allocation quantum) such as a word or a double-word. A bitmap is a vector of bitflags, with each bit corresponding to one quantum of memory. A bit value of 0 indicates a free quantum, while 1 indicates
an allocated quantum. The memory overhead — the space required to keep the bitmap itself — is modest. For example, if the quantum size is 32 bytes, the required bitmap size is 256 times smaller. Thus, to map 1 GB of space, the required bitmap size is just 4 MB. Given this example, to allocate an object of size S, the allocator must locate a contiguous area of free memory that contains QS=(S+31)/32 quantum blocks. The allocator does that by searching its bitmap to find a zero bits sequence QS long and turning over the bits once this sequence is found (Figure 3). One of the common ways to improve bitmap allocator performance is to search for free memory (a hole) by looking up the bitmap starting from the bitmap position where the previous lookup was left off, rather than from the beginning of the bitmap. When the allocator locates the free hole, the current bitmap position is updated. In addition to its speed advantage, this technique increases the locality of reference (objects allocated one after another are positioned sequentially in memory). Another way to speed up the bitmap lookup is to replace bit-by-bit scans with more efficient techniques. It is possible to scan the bitmap bit-by-bit (see Listing Four, available electronically). However, the bitmap can be more efficiently scanned one byte at a time using 256-way lookup tables to detect a free hole of the desired length (Listing Five, available electronically). The first table specifies the number of leading zeros in each byte; the second table represents the number of trailing zeros in each byte. The last two arrays specify the number and the offset of the longest sequence of clear bits in the byte. Using these tables, the allocator can scan the bitmap as in Listing Six (available electronically). Bitmap allocators have a number of advantages. One feature that is especially important in database development is that the
bitmap mechanism can allocate a group of objects sequentially, thus maintaining locality of reference. In database development, the increased locality of reference means that objects allocated sequentially would be placed on the same database page. Consequently, these objects would be loaded into memory using just one read operation from disk, in contrast to multiple reads that would be required to fetch the objects if they were located in different parts of the storage. For a main memory database, locality of reference is less critical — after all, memory is a randomly accessed device. However, modern systems often include a secondary cache (L2 cache) that benefits from increased locality of reference as well. Furthermore, the bitmap allocator can keep fragmentation at bay. When objects are allocated by a quantum that is comparable to their own size, small unused holes (such as those accumulated by linked-list allocators) are never created. Another performance advantage lies in the fact that bitmaps themselves are not interleaved with main storage, which improves the locality of searching. The locality of writes may also be improved because freeing objects only modifies the bitmap. Apart from database systems, bitmap allocators are quite commonly used in garbage collectors, database dump utilities, and filesystems’ disk block managers — areas where enforcing locality of reference and reduced fragmentation are imperative. Conclusion For developers of database systems, filesystems, compilers, and similar applications, creating custom memory managers rather than using an operating environment’s existing allocators entails extra coding — sometimes a great deal of it. But such allocators, once written, establish an infrastructure that permits optimization. DDJ
Listing One
Listing Two
template class fixed_size_object_allocator { protected: T* free_chain; public: T* allocate() { T* obj = free_chain; if (obj == NULL) { obj = new T(); } else { free_chain = obj->next; } return obj; } void free(T* obj) { obj->next = free_chain; free_chain = obj; } fixed_size_object_allocator() { free_chain = NULL; } ~fixed_size_object_allocator() { T *obj, *next; for (obj = free_chain; obj != NULL; obj = next) { next = obj->next; delete obj; } } };
class BlockAllocator { void* allocate(size_t size) { if (size + sizeof(object_header) <= page_size/2) { int n = block_chain[((size+sizeof(object_header)+7)>>3)-1]; storage_free_block* bp = hdr->free_block_chain[n]; if (bp != NULL) { hdr->free_block_chain[n] = bp->next; bp->size = size; return (object_header*)bp+1; } } // allocate a new block using some external allocator return alloc_block(size); }
http://www.ddj.com
void free(object* obj) { object_header* hp = get_header(obj); if (hp->size + sizeof(object_header) <= page_size/2) { int n=block_chain[((hp->size+sizeof(object_header)+7)>>3)-1]; storage_free_block* bp = (storage_free_block*)hp; bp->next = hdr->free_block_chain[n]; hdr->free_block_chain[n] = bp; } else { // return the block back to the memory pool free_block(hp); } } };
DDJ
Dr. Dobb’s Journal, December 2005
75
PROGRAMMING PARADIGMS
New Orleans Memories Michael Swaine What has happened down here Is the winds have changed Clouds roll in from the north And it started to rain — Randy Newman, “Louisiana 1927”
I
’ve only visited New Orleans twice, once at the terminus of a 4000-mile camping trip with Nancy and once with other Dr. Dobb’s staff for the fourth annual ACM conference on Object-Oriented Programming, Systems, Languages & Applications — OOPSLA ’89, in other words. One trip was in the year of the Oakland Hills fire, at that time, the worst urban disaster in U.S. history; the other in the year of the Loma Prieta earthquake. The two trips blur into one music- and food-suffused montage. Street scenes and snatches of Dixieland, faces and voices, all come back to me stripped of context: fragmentary images that bob to the surface and float away. A colorfully dressed stranger in the French Quarter offering to bet me that he could guess where I got my shoes. A young woman with braided hair sitting on the sidewalk on Magazine Street playing a saw. Tiny boys who don’t look like they’ve been walking all that long tap-dancing in the street as tourists weave around them. A trumpet player in a baby-blue sportscoat sitting across the table from me and talking about software development. The trumpet player is also a piano player, composer, ventriloquist, short-story writer, cartoonist, C programmer, Dr. Dobb’s columnist, and friend. Al Stevens calls Cocoa Beach, Florida, home — but either there or here in New Orleans, the babyblue sportscoat looks appropriate. Al’s in town for OOPSLA, too. I remember that dinner, and can almost pick out all the faces around the table. It was, like every meal I ever ate in New Orleans, memorable. Many of my memories of New Orleans involve food. Eating beignets for breakfast, getting powdered sugar in my beard. The best beignets, I’m told, are to be found at CDM, the famous Cafe Du Monde, operating in the French Quarter since 1862. Right on the bank of the Mississippi River. “Nowhere,” John McPhee wrote in an essay two years before that OOPSLA conMichael is editor-at-large for DDJ. He can be contacted at [email protected]. 76
ference, “is New Orleans higher than the river’s natural bank…The river goes through town like an elevated highway.” Chicory and Po-Boys In National Geographic And the Times-Picayune They forecast the apocalypse Said it was coming soon — David Rovics, “New Orleans”
I’m a coffee drinker and I want to like New Orleans coffee, which by unvarying tradition is always cut with chicory. Maybe it’s an acquired taste. Maybe it just doesn’t pack enough caffeine kick for my longestablished habit. As I am a reader by even longer-established habit, I read the street signs: Abundance, Pleasure, and of course Desire. There’s a bus line here called “Cemeteries.” I read the brochures, read that the chicory tradition began during the Civil War, when New Orleanians stretched war-scarce coffee with the roasted and ground root of the locally harvested endive plant in response to the Union blockade of the Port of New Orleans. In the century and a half since the Civil War, the Port of New Orleans had grown to become the largest port in the United States and the fifth largest port in the world. Until September 2005. I remember eating po-boys for lunch. Po-boys are something like submarine sandwiches, crunchy-crusted French bread with just about any filling, served plain or dressed. But it is specifically oyster poboys that I remember with such fondness. Seafood and fresh-water fish or shellfish are a big part of New Orleans cuisine. I remember snacking on crawdads spilled out on newspaper, taking a taxi to a dinner that I am pretty sure featured fish, in a good restaurant in an old mansion in the Garden District. I understand that this area was among the least affected by the storm and its aftermath. “In New Orleans,” McPhee wrote in 1987, “income and elevation can be correlated on a literally sliding scale: the Garden District on the highest level, Stanley Kowalski in the swamp.” Later, walking through the French Quarter with a drink in one hand and a snack in the other, between ornate balconies overhanging the sidewalks, surfeited but tempted by the good smells all around. New Orleans had more well-known chefs Dr. Dobb’s Journal, December 2005
than any other city its size, and more good places to eat. Now even famed television celebrity chef Emeril Lagasse’s three New Orleans restaurants are closed, his web site turned into an online contact point for refugee employees widely scattered in the New Orleans diaspora. Lost and Scared My memory is muddy, What’s this river that I’m in? New Orleans is sinking And I don’t wanna swim —The Tragically Hip, “New Orleans Is Sinking”
I remember checking into a French Quarter hotel on a hidden courtyard down some anonymous alley. Dinner that night in a restaurant so tucked away on an alley/ courtyard that I don’t know how we even found it. The live music may have led us there, although live music is like air in the French Quarter. The next day finding in a funky little curio shop a cleverly framed Picasso print that we had to have. Hidden treasures. Some of these memories are from that camping trip, others from OOPSLA. They run together like they’d been left out in the rain. This memory is definitely from the camping trip, though: That summer, we had come crosscountry in our camper van, arriving in New Orleans late in the evening. Looking for a campground in a black downpour, we ended up we-had-no-idea-where outside of town next to some body of water. Lake Pontchartrain? The Mississippi River? It was too dark and rainy to tell and the people running the place spoke a dialect of English we didn’t understand. Parked in our van a few dozen feet horizontally and maybe two feet vertically from the troubled surface of the dark water, thunder booming around us and the water rising, we didn’t get much sleep. In the morning, we realized that we had never been in any real danger. But the memory of that night comes back to me vividly these days as I think about the catastrophe of New Orleans 2005. OOPSLA ‘89 I added up the preregistration and on-site registration lists, and came up with 1626 registered attendees of OOPSLA ’89. You may have to just take my word for it that http://www.ddj.com
United States Postal Service Statement of Ownership, Management, and Circulation
this was a lot of people for an objectoriented programming conference back then. It was easy to conclude, I concluded, that there was an object-oriented programming revolution underway. Today, the object-oriented paradigm is taken for granted, and proponents of newer approaches like Aspect-Oriented Programming position their paradigm against it, like a candidate running against an incumbent. But in 1989, OOP was the coming thing. I thought it might be interesting to look back at some of the highlights of that conference at the New Orleans Hyatt Regency in the first week of October 1989. The exhibit floor showed which companies considered OOP important enough to rent booth space: Digitalk, The Whitewater Group, and Interactive Software Engineering had prominent placement, as did Apple and Borland. On the other hand, Microsoft didn’t exhibit, although they had plenty of employees at the conference. The conference itself was chaired by Kent Beck, currently a neighbor of mine here in Southern Oregon. Kent was with Apple then, and he and Ward Cunningham gave a talk on the teaching of OOP. Before the regular sessions there were two days of tutorials on what were considered the most important OOP environments, including Smalltalk, C++, NextStep, and a portable C++ class library for UNIX called ET++. The keynote address was by Peter Wegner, who wrote the first book on Ada and got into OOP because of his perception of the deficiencies of Ada for software engineering. He left little doubt that he considered software engineering to be the proper goal for OOP. Wegner’s latest book, out this year from Springer-Verlag, is called Interactive Computation: The New Paradigm. The ’89 conference marked a new stage of maturity for OOP: Kent Beck explained that, while previous OOPSLA conferences had included sessions on such related topics as general software engineering, user interface design, and databases, this year’s conference would focus more tightly on OOP theory, OOP language design and implementation, and concurrency. Class Warfare There’s a perpetual problem for those who would define and name paradigms in technology: As soon as you assign two or more attributes to a paradigm, someone will come up with an approach that includes all but one of the attributes. It was clear at that ’89 conference that there would be no precise definition of OOP that would encompass all the interesting work going on in the area. The SELF language, the subject of one session, is a good example. Can a language that has objects and inheritance but that lacks classes qualify as object oriented? http://www.ddj.com
Software engineering and the tantalizing vision of interchangeable software parts was the subject of a number of sessions. Brad Cox, who had written a new object-oriented version of C called “Objective-C,” led a discussion on what was optimistically called the “Software Engineering Revolution.” Brad’s still on the ramparts in that revolution, but he mostly programs these days in Java, Ruby, Python, and Perl, rather than his own language. A number of sessions described progress in real-world applications of OOP. I don’t know if you’d consider Star Wars, also known as the Strategic Defense Initiative, real-world, but other presentations addressed the application of OOP to practical issues in commercial software development, CAE/CAD, and scientific computing. The scientific computing papers suggested that most people working on OOP in this area were using either C++, Smalltalk, or CLOS, the Common Lisp Object System. These scientific sessions made it glaringly clear that the ability to construct 3D computer models of objects and processes and to manipulate the models had by then become an essential tool of scientific research. It was also clear that OOP solutions had some real challenges in efficiency and performance, as indicated by the fact that many systems dropped out to plain C code in critical sections, and by the sessions on the search for ways to speed-up garbage collection. Sun was there, too, of course. These days, Scott McNealy is Sun Microsystems’ point man for Infuriating People and Insulting Competitors. But in New Orleans that fall, it was Bill Joy saying that he didn’t consider traditional data-processing professionals a market for OOP because they wouldn’t understand it — and adding that, to be fair, he’d say the same about most C programmers. Some of the chief issues pertaining to pushing the state of the art that came up at the conference were persistent objects, parallelism, agents, and reflection. There are many kinds of software agents, but one particular kind was on people’s minds at the conference. Several talks dealt with developing agents to represent the user — or programmer or maybe the program — in searching for objects in some future world of sharable, reusable software components spread among widely distributed systems. Still something of a dream, but a little closer to reality today, maybe. During one of the panel discussions, C++ creator Bjarne Stroustrup said that anyone claiming that object-oriented programming will bring about bug-free software was probably spending evenings in Dr. Dobb’s Journal, December 2005
1. Publication title: Dr. Dobb’s Journal 2. Publication number: 1 0 4 4 ---- 7 8 9 X 3. Filing date: 10/1/05 4. Issue frequency: Monthly 5. Number of issues published annually: 12 6. Annual subscription price: $34.95 7. Complete mailing address of known office of publication: CMP Media LLC, 600 Harrison Street, San Francisco, CA 94107 8. Complete mailing address of headquarters or general business office of publisher: CMP Media LLC, 2800 Campus Drive, San Mateo, CA 94403 9. Full names and complete mailing addresses of publisher, editor, and managing editor: Publisher Michael Goodman, CMP Media LLC, 2800 Campus Drive, San Mateo, CA 94403; Editor-in-Chief Jonathan Erickson, CMP Media LLC, 2800 Campus Drive, San Mateo, CA 94403; Managing Editor Deirdre Blake; CMP Media LLC, 2800 Campus Drive, San Mateo, CA 94403 10. Owner, full name: CMP Media LLC. Complete mailing address: 600 Community Drive, Manhasset, NY 11030-3847. An indirect, wholly owned Subsidiary of United Business Media plc, Ludgate House, 245 Blackfriars Road, London, SE1 9UY UNITED KINGDOM 11. Known bondholders, mortgages, and other security holders owning or holding 1 percent or more of total amount of bonds, mortgages, or other securities: None 12. Not non profit: N/A 13. Publication title: Dr. Dobb’s Journal 14. Issue date for circulation data below: Average from July 2004 to June 2005 and October 2005 issue 15. Extent and nature of circulation: a. Total number of copies (net press run) Average number of copies each issue during preceding 12 months: 145,188. Actual number of copies of single issue published nearest to filing date: 142,924. b. Paid and/or requested circulation. (1) Paid/requested outside-county mail subscriptions stated on form 3541. Average number of copies each issue during preceding 12 months: 108,538. Actual number of copies of single issue published nearest to filing date: 114,186. (2) Paid in-county subscriptions stated on form 3541. Average number of copies each issue during preceding 12 months: 0. Actual number of copies of single issue published nearest to filing date: 0. (3) Sales through dealers and carriers, street vendors, counter sales, and other non-USPS paid distribution. Average number of copies each issue during preceding 12 months:14,342. Actual number of copies of single issue published nearest to filing date: 11,497. (4) Other classes mailed through the USPS. Average number of copies each issue during preceding 12 months: 0. Actual number of copies of single issue published nearest to filing date: 0. c. Total Paid and/or Requested Circulation [Sum of 15b. (1), (2), (3), (4)]. Average number of copies each issue during preceding 12 months: 122,880. Actual number of copies of single issue published nearest to filing date: 125,683. d. Free distribution by mail (samples, complimentary, and other free): (1) Outside-County as Stated on Form 3541. Average number of copies each issue during preceding 12 months: 0. Actual number of copies of single issue published nearest to filing date: 0. (2) In-County as Stated on Form 3541. Average number of copies each issue during preceding 12 months: 0. Actual number of copies of single issue published nearest to filing date: 0. (3) Other Classes Mailed Through the USPS. Average number of copies each issue during preceding 12 months: 0. Actual number of copies of single issue published nearest to filing date: 0. e. Free distribution outside the mail (carriers or other means). Average number of copies each issue during preceding 12 months: 2,510. Actual number of copies of single issue published nearest to filing date: 2,872. f. Total free distribution (Sum of 15d and 15e). Average number of copies each issue during preceding 12 months: 2,510. Actual number of copies of single issue published nearest to filing date: 2,872. g. Total distribution (Sum of 15c and 15f). Average number of copies each issue during preceding 12 months: 125,390. Actual number of copies of single issue published nearest to filing date: 128,555. h. Copies not distributed. Average number of copies each issue during preceding 12 months: 19,799. Actual number of copies of single issue published nearest to filing date: 14,369. i. Total (Sum of 15g and 15h). Average number of copies each issue during preceding 12 months: 145,189. Actual number of copies of single issue published nearest to filing date: 142,924. j. Percent paid and/or requested circulation. (15c divided by 15g times 100). Average number of copies each issue during preceding 12 months: 98%. Actual number of copies of single issue published nearest to filing date: 98%. 16. Publication on statement of ownership. Publication required. Will be printed in the December 2005 issue of this publication. I certify that all information furnished on this form is true and complete.
Michael Goodman, Publisher
77
the French Quarter guessing where people got their shoes. I guess he ran into that guy, too. Katrina Mashups The online community— and this is one of those cases where that expression is not an
oxymoron — responded quickly and in many ways to hurricane Katrina. Among those responses were some creative applications of map-based mashups, including visual plots of Craigslist refugee housing locations and maps showing information about affected locations in New Orleans.
Links to blog topics are often ephemeral, but http://googlemapsmania.blogspot.com/ is at least a starting point for reading about mashups and for tracking down some of these clever and useful hacks. DDJ
Dr. Ecco Solution Solution to “Feedback Strategies,” DDJ, November 2005 1. The probability of winning under FeedYes is about 0.89 when Pgood is 0.9. It cannot be better than 0.9 because if one is on the second from top row and is adjacent to the winning square, there is some chance of losing. The strategy for FeedYes consists of trying to reduce the horizontal distance to the goal to zero (or to one on rows that are an odd distance away). Calculating the probability involves the following key recurrence ideas: If you are n steps (n>0) away from the top row and 0 distance away, then your probability of hitting is the same as the probability of hitting if you are n–1 steps away and one away from column 5. Otherwise, with prob-
ability Pgood you will be one column closer to your goal and with probability 1–Pgood you will be one farther from the goal both with n–1 steps. Under the FeedNo strategy, the probability is about 0.55. To compute the probability for FeedNo, assume a strategy that will take four aims to the right and three aims to the left since this gives the best likelihood of hitting the destination. Given this, there are four ways to win: All seven aims are true; three of the right aims are true and two of the left are; two of the right are true and one of the left; and one right and zero left. So, the feedback dividend is about 1.6. 2. The value of Pgood, for which the dividend ratio is highest, is not 0.75 as one might think but something around 0.772.
For that value, the dividend ratio reaches 1.99. I don’t believe it ever reaches 2. 3. In both the FeedNo and FeedYes cases, the analysis combines short trips: One wants to go from row 1 column 4 to row 3 column 4 in the first two moves then to row 5 column 4 in the second two moves, then row 7 column 4, and finally row 8 column 5. For FeedNo, this gives a formula like the following: ((((Pgood 2)+(1–Pgood )2))3)×Pgood For FeedYes, this gives a formula that is Pgood 4. This gives a maximum feedback dividend of 1.75 when Pgood is 0.71. Alan Dragoo helped with these solutions. DDJ
Dr. Dobb’s Journal “Wit” T-Shirt Contest Dr. Dobb's Journal is creating some brilliant and hilarious t-shirts and/or bumper stickers for software developers. Here's the catch: we haven't actually come up with any ideas yet. We're counting on YOU, the readers of Dr. Dobb's Journal, to help us out with some killer slogans. We are interested in original, and wit, developerrelated slogans. If you can't think of any topics yourself, here are some suggestions: software architecture specific languages or platforms Quick. Birth- open source -how how developers relate to the world -Accurate. the world relates to developers – debugging –School security - development teams Inexpensive.
And the winners are...
If we choose one or more of your submissions, we'll print your name and Pick two winning slogan in Dr.Death Dobb's Journal and send you five samples of the product(s) we create (total monetary value, $5).Terwilliger To enter, please send your Nancy slogan(s) to: [email protected] by midinight P.S.T. May 31, 2005.
Dr.only Dobb's Remember, thisBecause could be your claim tosaid fame ...so. well, at least among your colleagues! Remember,Stephen while the prize value may only be five Craver bucks, this could be your only claim to fame …. well, at least among your peers!
By submitting your ideas to the Dr. Dobb's Journal “Wit” T-shirt Contest, you are granting Dr. Dobb's Journal full rights to your submission, including the right to give away or sell products with that include the slogan displayed upon them. No purchase necessary. Void where prohibited. Contest open to residents of the 50 United States including the District of Columbia (excluding Florida residents) and Canada (except Quebec) who are 18 years or older. See official rules for full details at www.ddj.com/contest.html/
78
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
EMBEDDED SPACE
Linux Symposium 2005 Ed Nisley
symposium (plural: symposia) 1. A conference or other meeting for discussion of a topic, especially one in which the participants make presentations 2. (in ancient Greece) a drinking party, especially one with intellectual discussion
—Wiktionary
T
he 7th Annual Linux Symposium took place in Ottawa. In addition to formal presentations and tutorials, informal Birds of a Feather Sessions covered specific topics of interest to small groups. Unlike larger trade conferences, the Linux Symposium (LS) has no exhibit floor, no commercial presence, and no hustle. This makes for a much smaller and quieter show, as the attendees come for education rather than entertainment. Well, at least during the day, as the WhiskeyPurchasers BoFS met every evening. I was impressed by the intensity of the presentations and eventually figured out what’s different. Unlike some speakers at commercial conferences, these folks haven’t been picked for their speaking ability or stage presence. They speak from deep, first-hand knowledge, rather than reciting bullet items generated by someone else. In fact, they may be the “center of competence” for one particular part of the Linux system. The audience also carries knowledge and interest that transcends language barriers. A question from the ranks often triggered an esoteric discussion between people who obviously spoke English as a Ed’s an EE, PE, and author in Poughkeepsie, NY. Contact him at ed.nisley@ieee .org with “Dr Dobbs” in the subject to avoid spam filters. http://www.ddj.com
second or third or fourth language, but who had no trouble at all communicating. It was refreshing, to say the least. The Big Picture The single most significant thing I observed throughout all four days was that Big Business is now pushing Linux development hard. Perhaps the best indication of that can be seen in the companies employing the presenters. To judge from the 53 Linux Symposium speaker biographical sketches, IBM sent 17 speakers, Intel 8, HP 3, Red Hat 3, and other Linux distros about 10. Yes, half the speakers work at various IBM, Intel, and HP locations around the globe. In contrast, the 43 speakers at the 2nd Annual Linux Symposium in 2000 came from smaller operations. LinuxCare sent 7, Red Hat 5, VALinux 4, ZeroKnowledge 3. On the big-company end, IBM and HP sent one speaker apiece. As you might expect, big-company programmers work on big-company problems. That’s not to say the rest of us don’t benefit from the work, but you probably don’t do a lot of memory hot-plugging, either. The era of the lone, unsupported coder seems to be drawing to an end, if only because increasing complexity requires a team of experts. In fact, mere mortals can’t afford the gargantuan hardware that exercises the new features. Of those 53 speakers, about 38 gave talks on kernel or system topics. The remaining presentations covered a wide variety of business, application, and embedded topics. The Linux kernel and its infrastructure are undergoing heavy development, with extensions in a number of useful directions. Dr. Dobb’s Journal, December 2005
Shrinking the Large Given that many of the LS papers deal with large-scale kernel topics, what’s in it for embedded systems? Jonathan Corbet (Linux Weekly News) said it best in his keynote address: “Today’s big iron is tomorrow’s laptop.” As we’ve seen, today’s laptop becomes next week’s embedded system. The majority of embedded systems run 8-bit microcontrollers that simply lack the moxie for Linux. The small slice of the market that can afford the megabytes of memory and megahertz of speed required to run a complex operating system currently supports a mix of commercial RTOS and Linux vendors, plus the usual homegrown solutions. Within that slice, commercial RTOS vendors continue to hold their own in those projects requiring software behavior certified to specific safety and operational standards. Most systems don’t require that level of certification, however, providing a perfect entrée for Linux. As nearly as I can tell from the few numbers I’ve seen, the growth rate for embedded Linux is far higher than anything else and the absolute usage is at least approaching that of traditional RTOS designs. Although it’s an oversimplification to say that any gadget that can run Linux will run Linux, that’s probably not far from the truth. The Consumer Electronics Linux Forum exhibited several such trinkets at its CELF BoFS, including a minuscule mobile phone camera. Problems remain with memory footprint, power consumption, and real-time performance. The examples showed that good-enough solutions are available now and better ones are pending. 79
Suresh Siddha (Intel) described the changes required for a “Chip Multi Processing Aware Linux Kernel Scheduler.” The current scheduler ably wrings maximum performance from multiple separate CPUs, but lacks the information to meet other goals that will become critical in the giga-transistor CPUs now looming over the horizon. Up to this point, maximum-performance scheduling simply meant keeping each CPU busy while the hardware sorts out nuances such as cache contention and bus bandwidth. That works well for CPUs optimized for single-instruction-stream processing. Although Intel’s Hyper-Threading technology made one physical CPU look like a pair of CPUs sharing one memory interface, memory contention could affect overall performance. When Intel’s Pentium roadmap imploded due to terrible power-versus-performance numbers, the ensuing multicore CPUs vastly complicated the memory interfaces. System boards with multiple chips, each with multiple CPU cores, each capable of multithreaded execution, can have spectacular performance, if only we figure out how to take advantage of it. A multicored, multithreaded gaggle of CPUs can achieve high throughput with a slower clock, simply because more results emerge per tick. That’s assuming enough memory bandwidth to keep the pipes full, software amenable to parallel execution, and a scheduler competent to orchestrate the whole affair. What’s new and different is the requirement to simultaneously minimize power consumption for a given software-performance level, with the upper power limit and the lower performance limit chosen on the fly. Power consumption in CMOS circuits varies almost linearly with the hardware clock speed, so slowing a core’s clock linearly reduces its power consumption, shutting off the clock to unused hardware helps, and stopping a core drops its power consumption nearly to zero. Unfortunately, the vagaries of chip design may force different cores to share a single power source. The lowest overall power may therefore require clocking a single core at top speed, slow-clocking two cores sharing a power source, or some even more bizarre combination. The rules depend on the exact chip, so the scheduler must use deep hardware knowledge that’s currently unavailable. The eventual solution will involve percolating power and clock rules up to the scheduler so that it can determine the cost of various policies, rather than simply keeping all of the hardware busy all the time. Although this talk concerned Intel CPUs, various DSP and embedded-processor 80
vendors have introduced multicore chips in the last year or so. Most of these appear in applications that require both high performance and low power, so the trend is clear: The end of single-threaded CPU hardware is at hand. It’s not clear just how many embedded applications can realize a major performance boost from hardware multiprocessing, but maybe yours will. Getting It Right When the Linux kernel was the exclusive domain of enthusiasts, reliability and stability were less important than bare functionality. Big businesses with the Linux kernel at the core of their operation have begun placing major emphasis on getting the bugs out of both existing code and new functions. Rusty Russel (IBM) described “nfsim: Untested Code Is Buggy Code.” The project resurrected the netfilter test suite from the bitrot morass, created a more comprehensive test-case collection, and rebuilt the testing environment. The netfilter code manages the kernel’s network interface by massaging incoming and outgoing packets, so this code is at the heart of any networked system. The nfsim program allows “netfilter developers to build, run, and test code without having to touch a real network, or having superuser privileges.” In effect, nfsim is a virtual environment running actual kernel code with the ability to inject errors into simulated network traffic and observe the results. It is now feasible to test error paths in a way that simply isn’t possible on a real network. Injecting and tracking errors is expensive, turning a five-second mainline test run into a 44-minute exhaustive- error marathon. While developing the simulator and the test cases, they also flushed many longstanding bugs out of the netfilter code. In fact, one run using Valgrind to verify their own memory-allocation code lasted 18.5 hours, but exposed a kernel bug. Despite testing all the error paths, the total test coverage hovers around 65 percent. The tested code includes otherwise untestable paths as well as mainline code and provides considerable confidence that actual errors will be reported correctly. The programmers observe that any code lacking test cases almost certainly has errors, while noting that “developers have a certain antipathy to testing.” What’s needed is a change in mindset and they hope nfsim will provide an example of how to go about getting the job done. Hardware drivers form a particularly sore point for testing and verification. The central problem is that Linux supports a tremendous amount of hardware that doesn’t exist in any one place. A develDr. Dobb’s Journal, December 2005
oper working with, say, a new SCSI driver simply cannot test any existing code for Other People’s Hardware. As a result, old code suffers bitrot that may not be seen until the single organization owning that hardware installs the new code. You can see a certain circular pattern lying in wait, can’t you? The only solutions seem to be heaving out truly obsolete hardware features and simplifying the remaining code to the point where it must work. That Quixotic process is ongoing. Embedded Machinations The new verb “to brick” has been making the rounds, referring to a flash-memory update rendering a device unbootable, whether due to a power failure or a firmware error. Gilad Ben-Yossef (from Codefidence, as in “Code Confidence”) described cfgsh, a shell environment designed for safe in-the-field flash memory updates. It seems that embedded-system designers take the path of least resistance when planning update procedures. UNIX expatriates tend to manually untar archives and twiddle rc files, a process inevitably leading to completely inconsistent and undocumented firmware states, the bane of tech-support desks everywhere. RTOS veterans produce each firmware build as a huge binary lump, with any subsequent changes applied as a delta to the unpacked files. Regardless of the technique, an untimely reset can trash the firmware and leave the system unbootable: a brick. The cfgsh firmware automates the update process by completely configuring the new firmware, then flipping from the old firmware to the new in a single atomic operation. It also gracefully handles the case of continuous reboots or hangs due to firmware errors. In short, even if you think you can do this stuff yourself, read the paper to discover several failure modes you didn’t consider. Keith Packard (of HP, but probably no relation) described TWIN, a 100-KB Tiny WINdowing system that replaces the 4– 5 MB X Window system, in a talk that I couldn’t attend. TWIN is designed for “subPDA” devices such as phones and a cute distributed computer node in use at HP Cambridge. The secret to size reduction lies in a relentless concentration on not including every bell and whistle found useful in the last four decades of graphics research. Chasing the Kernel Dave Jones (Red Hat) presented the final keynote address on the need for better kernel bug reporting and testing, observing that, although the Linux kernel has an enviable reputation for continuity, it’s certainly not error free. Perhaps the fundamental http://www.ddj.com
problem is that kernels can’t be tested under real-world conditions until they’re released for general use, but for some unknown reason, nobody wants to run development kernels on production machines. Version-to-version kernel stability and backwards compatibility is pretty much ruled out by the Linux kernel development methodology, which is the main reason the kernel now accommodates such a wide range of hardware. However, internal great leaps also affect external interfaces, which poses a considerable problem for applications with projected lifetimes longer than Linux has been around. The best advice I’ve read is that embedded systems developers using the Linux kernel must break a long-standing habit by not keeping up with the latest kernel changes. Regardless of how nifty a new feature might be or which bug got squashed, you must learn to ignore those changes if they don’t affect your application. Of course, that means you must track and evaluate all kernel changes. But you were doing that anyway, right? Reentry Checklist The Linux Symposium differs from commercial conferences in another regard: You
http://www.ddj.com
can freely fetch the proceedings as PDF files from http://www.linuxsymposium .org/2005/.
“A multicored, multithreaded gaggle of CPUs can achieve high throughput with a slower clock” You can look up odd words in Wiktionary, a multilingual dictionary at http:// www.wiktionary.org/. If that’s too tame, try http://www.urbandictionary.com/. Ottawa also hosted a related pair of two-day Linux conferences prior to the LS: the Kernel Summit and the Desktop Conference. The former, a small invitationonly event for kernel developers, gathered key folks together in one place at
Dr. Dobb’s Journal, December 2005
one time for the sort of discussion, planning, and schmoozing you can’t do electronically. The latter concentrated on issues relevant to application development for the desktop, with a smattering of kernel issues. I didn’t attend either meeting, but you can read more at http:// lwn.net/Articles/143649/ and http://www .desktopcon.org/. A comprehensive comparison of Windows and Linux metrics, including some embedded OS design estimation, is at http:// www.dwheeler.com/oss_fs_why.html. The Valgrind project may be just what you need to get bugs out of your own code. It’s named after the entrance to Valhalla, pronounced “val- grinned,” and found at http://www.valgrind.org/. Subscribe to the Linux Kernel Mailing List at http://lkml.org/. Reader Motti Shimoni sends in a plea for you to turn off your monitor. He works in a huge corporation that does third shift remote updates, so all PCs must be left on overnight. It seems everyone also leaves the monitor on and nobody uses screensavers with display power management. Maybe that corporation lacks a clue, but yours shouldn’t: Turn on power management and turn off the monitors, okay? DDJ
81
CHAOS MANOR
Windows Vista: First Impressions Jerry Pournelle
W
e’ve installed Longhorn. Microsoft insists we call it Windows Vista, but most of the information screens still call it Longhorn. I have been told there will be seven versions of Vista when it is released. Phrases come to mind, such as the Seven Pillars of Vista, Seven Dancing Vista Masters… Installation beta proceeded smoothly. Our only problems were with hardware. We installed this on the test machine that had been running 64-bit Windows, and alas, that machine had been mined for memory and some other parts, and had to be reassembled. Then one of our highspeed video cards developed a fatal error. The card is fine, but the chip fan on the video board died, and it takes only seconds before the card ceases to output coherent video without the fan. I’ll replace the fan, but for now we have replaced the video board. Once we had a stable system, there were no further problems. We let the new OS reformat the hard drive, which is a pair of Seagate Serial ATA 160-GB Barracuda drives. The formatting took about 10 seconds, so clearly it employed fast format, not the more thorough formatting that inspects the drive surface. We were not offered that option; I presume that will be built into the installer in the final versions. After that, installation was normal, except for drivers. Our first trip was to the Windows Update site, where we were told that this site doesn’t support Longhorn systems. Then we went to nVidia where we got the very latest beta Vista drivers; the “very latest” don’t work and produced one of our system crashes. Doing a rollback in Vista got the video system working but with an amusing (amusing now, infuriating at the time) quirk I’ll tell you about later. Going to the motherboard site got us the latest Ethernet card drivers, but note that we were able to get on the Internet immediately with the generic drivers that came with Longhorn. Nvidia’s Vista beta sound drivers installed without problems, and in less than Jerry is a science-fiction writer and senior contributing editor to BYTE.com. You can contact him at [email protected]. 82
an hour from when we began we had an operating Longhorn system. When it came time to name the system, I called it “Servilia” for no particular reason — it’s not a server. The name came to me from the HBO series Rome, which, incidentally, I like quite a lot. I began playing around with Longhorn, and discovered to my horror that there was no “Start” button. That got me spending time learning things, such as that the F3 key brings up interesting search windows that don’t work quite the way you expect them to. Eventually, I was able to open a “Computer” window (having fruitlessly searched for “My Computer”) and use that to find some other programs, but nary a “Start.” This was first amusing, then frustrating, finally infuriating. About then Chaos Manor Associate Dan Spisak came over and experimented until he discovered that Vista was both displaying the wrong screen resolution and rendering that resolution incorrectly. This had the effect that the system thought the Start menu was on another screen when in fact it wasn’t displaying on the only screen. Once this was fixed I had a normal Windows Vista to play with — there really is a Start button, as you would expect — and I fooled around with it for a few hours. My initial impression is that I am going to like Windows Vista a lot. Remember, It’s Beta Having said that, let me warn you: This is very much a beta. We have managed to crash it several times, including at least one blue screen from which no recovery was possible short of turning off the system. We have also had crashes that took out the current processes but left the rest of the operating system intact. We have had instances in which processes simply could not be killed short of reboot. You would be wise to install beta Vista on a system whose contents you don’t care much about so that you can test the applications you use frequently without fear of data loss. Much of the user interface is unfinished. As an example, the wizard for connectDr. Dobb’s Journal, December 2005
ing a Longhorn system (again, Microsoft calls it Vista, but most of the internal information screens talk about Longhorn) to an Active Directory Microsoft Windows Network does not work, nor do a couple of other approaches to this elementary task; but once you cotton on to using the old-fashioned approach of right click on My Computer, Properties, Computer Name, and going on from there, connection to a network is quite painless and the networking works smoothly and swiftly. Of course, you need to know that My Computer is now just “Computer,” but you’ll discover that soon enough. And, of course, it’s still sometimes “My Computer” in some of the help documents, and if you go into various System Tools it’s “My Computer,” but we can be certain that Microsoft will clean up all that gubbage long before this becomes a release candidate, let alone before it gets out to the public, just as we can be sure they will put in all the help files. At the moment it can be disconcerting to find a help topic only to discover that the file consists of a placeholder and a promise to have an actual help file later. In a word: It’s a beta, and you would have to be mad to entrust any critical operations to it. On the other hand, it’s different enough from Windows XP that large establishments will probably want to get it running for at least one guru to play with, because I have a feeling this is going to be pretty popular when it finally comes out. Random Observations This is very much a first look, and I can’t do more than give impressions. I’ll have more in future columns. Drivers and peripherals work. After we had the system running, I installed a Plextor PX716A DVD Read/Write drive. The system recognized it instantly, and it’s ready to read and write DVDs. Nero 6 installs and runs properly, or you can use the builtin Microsoft Windows capability. Our initial system has “only” 512 MB of Kingston PC3200 DDR-SDRAM. Microsoft recommends 512 MB minimum, but for our tests this seemed to be plenty. There http://www.ddj.com
are three memory slots on the motherboard, and we expect to put in two Kingston 1-GB PC3200 DDR-SDRAM memory cards for a total of 2.5 GB, but I don’t have the new memory yet. With only 512 MB, the user interface is crisp and reasonably intuitive. Microsoft recommends a minimum of 1 GB for 64-bit Vista, with 2 GB recommended. Memory is cheap, and that’s just as well. Programmers will add features and expand requirements to use all the memory you can give them. That’s often a criticism, but in fact it can be a good thing. Code kludges to save memory have led to some notorious bugs. We don’t know just how much of the Longhorn Aero feature set has been incorporated into the beta; so far nothing spectacular beyond semitransparent windows (the transparency is adjustable, and works almost exactly like the transparency settings of dialogue and command windows in Everquest II). Vista is pretty cool. Of course, in the coolness department, it’s playing catch-up with Mac OS X 10.4 Tiger, which is not only cool but also stable and shipping. It’s also playing a bit of catch-up to desktop Linux, as Bob Thompson points out. KDE already has much, if not all, of the Vista eye candy, and Reiser4 has a filesystem better than what we expect Vista to ship with. For most of its life Microsoft has always run scared. That is in Bill Gates’ nature, and of course he was one of the few people to not only understand the implications of Moore’s Law, but to act on them. Most of Microsoft’s success has been due to anticipating hardware gains, which led to the philosophy of “ship it soon; the hardware will bail you out.” The big attractiveness of Vista will be its Windows applications compatibility, better search and file-organizing capabilities, and vastly improved security. Mac users can point out that except for the Windows applications compatibility, they already have all that, and for that matter, Macs run a perfectly good version of Microsoft Office that will create files you can open in Windows or Mac systems. I decline to get into that battle. I like small computers, I’ve always had at least one good Apple system, and if Apple made a good TabletPC, I’d seriously consider going over to the Mac just because on the Mac everything is either very easy or impossible (and with OS X 10.4 much of the impossible becomes merely as difficult as nearly everything is with Windows). Having said that, I also have to say that I am used to Windows. It’s familiar and easy enough for me to use — the difficulties come when I am trying to explain to someone else how I do something — and I haven’t had a security problem for http://www.ddj.com
more than 10 years. And Windows has two big things going for it — games, particularly massive multiplayer online role playing games (Everquest II if you must know), and TabletPC OS and OneNote. A Mirra Report Mirra (http://www.mirra.com/) is a Linuxbased backup server system, based on reports from a reliable reader who had installed it in his clinic. I now have the Mirra system in place, and I can say it works very much as described.
“My initial impression is that I am going to like Windows Vista a lot” Installation is simple with one problem — the instructions tell you to wait for an orange light. Either that’s wrong, or the orange LED on my Mirra system is defective. This turns out to be unimportant because the orange light is simply a signal that certain operations have been completed, and there are many alternative ways to determine that. The beauty of the Mirra backup system is that, once it is in place and running, you never need think about it. We have in fact tested its capability to restore some crucial data files, and it works exactly as advertised. Other than that, I often forget that Mirra is here at all. It sits back in the cable room and sucks in data from four different networked machines, and It Just Works. In particular, Mirra grabs the open outlook.pst file from Anastasia, my main communications workstation. Of course, that file is huge and changes all the time, so Mirra is often well behind; but because it is not overwriting its previous backup, there is always a more or less up-to-date backup file, and eventually there will be a quiet enough period that Mirra can catch up on my 2-GB outlook.pst file. There is one glitch. Mirra believes that the Chaosmanor web folder where I keep the working files for The View From Chaos Manor (http://www.jerrypournelle.com/) holds system files. Nothing I can do manipulating file attributes can disabuse Mirra of that notion, and it will not attempt Dr. Dobb’s Journal, December 2005
to backup anything in its contents. Now by its very nature, a web source file has a backup out on the web server, but this inability is annoying. I have reported it to the Mirra people and they say the next revision of their software will fix this problem. In the meantime, I run xcopy from a command line with the /e/s/d/y switches to copy the Chaosmanor folder off to our “box of drives” backup storage box. I know there are many Linux-based backup systems, and several readers have written to recommend theirs. I have tried to open communications with some of those companies, but so far to no avail. Meanwhile, I know the Mirra system works as advertised with the one glitch about thinking that webs are system files. It’s the simplest and most easily installed backup system I’ve run across. I have the 400-GB model, and I wonder if that’s too small, but it hasn’t proved to be. Recommended. Winding Down The game of the month is Microsoft’s Dungeon Siege II, which is an improvement over the already enjoyable Dungeon Siege I. You can play Dungeon Siege alone, in a LAN party with friends, or online with either friends or a pickup group, and it’s as good a way to while away an afternoon as any. My preference in online games remains Everquest II, but then I have a lot of time and energy invested in that. I used to like Star Wars Galaxies a lot but they kept tweaking it until it really wasn’t so much fun any longer, at least for me. In any event, you won’t regret getting Dungeon Siege II, although you may find yourself spending too much time investigating one more place; it’s very seductive that way. There are two Computer Books of the Month. Maven: A Developer’s Notebook, by Vincent Massol and Timothy O’Brien (O’Reilly & Associates, 2005), is about the Maven Java “project comprehension tool” for building your project. Chances are if you don’t work on Java projects (and perhaps even if you do), you have never heard of Maven; which is one reason to look into this book, if only to see whether you ought to learn more. Second is Brian Hook’s Write Portable Code: An Introduction To Developing Software for Multiple Platforms (No Starch Press, 2005). I am neither a user nor an admirer of any variant of C — I remain stubbornly convinced that strongly typed languages with range checking make for far better code with far less debugging — but experienced C users advise me that I will not go wrong by recommending this book. DDJ 83
PROGRAMMER’S BOOKSHELF
Six Books, Seven Paragraphs Gregory V. Wilson
I
am clearly obsolete. Not only do I prefer e-mail to instant messaging, I have a first and a last name, both spelled conventionally, and neither containing any nonalphabetic characters. Recycle me now. But while you’re waiting for the big green truck to arrive, have a look at 37signals’ Defensive Design for the Web. It (They? Pronouns have become so tricky) is/are a Chicago-based design firm, and this book is what they know about designing web sites that gracefully handle missing pages, out-of-stock items, and glitches in multistep registration processes. Its/their lean writing, clean layout, and carefully explained examples earned a double handful of yellow sticky notes, and a “must read” recommendation. Ian Langworth and chromatic-with-asmall-c’s Perl Testing is just as well written, but aimed at a much smaller audience. As the title suggests, its focus is how to test, and test with, Perl. There’s lots of good stuff here, and I particularly liked the organization: Each topic includes “How Do I Do That?”, “What Just Happened?”, and “What About…?” sections. However, you have to be a fairly strong Perl programmer to make sense of it all — I used to teach the language, but still found myself lost in the forest at least once a chapter. Next up is Jared Richardson and William Gwaltney’s Ship It!, which is a “lite” version of Hunt and Thomas’s instant classic The Pragmatic Programmer, with bits and pieces of other Pragmatic books stirred in. (Fair notice: The Pragmatic Bookshelf published my last book, too.) Everything you’d expect is here, from the importance of version control and an automated build-and-test cycle, to “tracer bullet” development and advice on how to handle common problems. None of it is new, but it’s all good stuff that the world needs more of. Greg is a DDJ contributing editor, and can be reached at [email protected]. His most recent book is Data Crunching: Solve Everyday Problems Using Java, Python, and More. http://www.ddj.com
Defensive Design for the Web 37signals, Matthew Linderman, and Jason Fried New Riders, 2004 246 pp., $24.95 ISBN 073571410X Head First Servlets & JSP Bryan Basham, Kathy Sierra, and Bert Bates O’Reilly & Associates, 2004 886 pp., $44.95 ISBN 0596005407 DHTML Utopia: Modern Web Design Using JavaScript & Dom Stuart Langridge SitePoint, 2005 318 pp., $39.95 ISBN 0957921896 Perl Testing: A Developer’s Notebook Ian Langworth and chromatic O’Reilly & Associates, 2005 180 pp., $29.95 ISBN 0596100922 Ship It! A Practical Guide to Successful Software Projects Jared Richardson and William Gwaltney Jr. The Pragmatic Bookshelf, 2005 198 pp., $29.95 ISBN 0974514047 Spring Into Technical Writing for Engineers and Scientists Barry J. Rosenberg Addison-Wesley, 2005 352 pp., $29.99 ISBN 0131498630 The fourth book this month is Stuart Langridge’s DHTML Utopia. I picked it up primarily because of the “Covers Remote Scripting/AJAX” splodge on the front cover. While there’s actually less than a dozen Dr. Dobb’s Journal, December 2005
pages on that particular topic, the rest of the book was well worth reading. After a quick introduction to how dynamic HTML works, Langridge shows how to make pages come alive with JavaScript. Animation, form validation, and yes, the asynchronous client/server communication promised in the splodge are all here, carefully dissected and clearly explained. I particularly liked the way this book shows the whole of each code sample first, then repeats snippets while explaining how they work, instead of the other way around. Bryan Basham et al.’s Head First Servlets & JSP is as thick as the previous four books are thin, but don’t let that stop you from buying it. At first (or even second) glance, the kung fu movie stills and art school layout make it look like a “for morons” title, but there’s a lot of hard-core technology in its almost 900 pages. The book covers all the topics in Sun’s Certified Web Component Developer exam: servlet configuration and lifecycle, custom tag development, a handful of design patterns (including Model-View-Controller), and a brief taste of the Struts framework. The book’s frenetic energy wore me out after a while, but I learned a lot from it, and my students (who are 20 years younger than me) have nothing but good things to say about it. Finally this month is Barry Rosenberg’s Spring Into Technical Writing for Engineers and Scientists. Rosenberg teaches the subject at MIT, and once you’re past the probably-worked-better- in-person jokes in the first couple of pages, it’s clear that this material is based on first-hand experience. The first major part of the book covers general principles of grammar, organization, and so on; the second analyzes particular kinds of documents — manuals, web sites, proposals, PowerPoint presentations, and so on — while the third describes the editing and production process. There’s a lot of crunch in this granola, and I’d recommend it to anyone with a thesis or a business plan to write. DDJ 85
OF INTEREST
Excel Software is starting MacTranslator OSX 1.0, a reengineering tool for generating graphic models of programming source code on Mac OS X computers. MacTranslator scans source code to extract design information that it outputs to a text file. That text is imported into the MacA&D modeling tool to automatically generate UML class models, structure charts, and rich data models. MacTranslator lets you generate accurate models of unfamiliar code in minutes. Through a mix of options and translation commands, MacTranslator can process over a hundred programming-language dialects of C, Pascal, Basic, Fortran, C++, Java, Object Pascal, Delphi, Objective-C, and SQL. Excel Software 19 Misty Mesa Court Placitas, NM 87043 505-771-3719 http://www.excelsoftware.com/ Xamlon’s Web lets you use C# or Visual Basic.NET in Visual Studio to build WinForms applications and deploy them directly to the Web without requiring the .NET runtime system. The product includes complete integration with Visual Studio.NET, and lets you use existing APIs you are already familiar with, including WinForms, GDI+, and XML web services. Programs are written in Visual Studio, then, from Microsoft’s Intermediate Language bytecode, Xamlon creates compact Macromedia Flash (SWF) files. The resulting applications can be deployed anywhere Flash is installed, including Windows, Macintosh, Linux, and PDAs and cell phones. The .NET runtime is not required for deployment and the resulting application may be deployed to platforms where .NET is not even supported. Xamlon Inc. 8899 University Center Lane, Suite 330 La Jolla, CA 92122 858-526-0704 http://www.xamlon.com/ http://www.ddj.com
Dekart has released Dekart Private Disk SDK, an SDK that lets you integrate on-thefly encryption, disk-exclusive access, and other Private Disk capabilities into any application. Dekart Private Disk SDK includes everything you need to build security into new or existing publishing, health care, legal, financial, and other applications. With the Dekart “disk firewall” feature, you can ensure that applications have an exclusive access to the protected disk, thus providing the highest possible level of security. The integration of “disk firewall” allows developers to rest assured that all the intellectual property created with their applications will never be disclosed to any unauthorized third party. The API included in Dekart Private Disk SDK can be transparently integrated into existing programs, avoiding the need to rewrite the interface or the existing documentation. Dekart, s.r.l. 75, Alba Iulia MD2071, Chisinau Republic of Moldova 373-22-245580 http://www.dekart.com/ Franson Technology has released Franson CoordTrans 2.0, a tool for converting geographic coordinates. A coordinate from a GPS typically uses the WGS84 datum, but maps and many GIS applications use other coordinate systems like NAD83 and UTM. CoordTrans can be used to convert between virtually all coordinate systems on Earth. Franson CoordTrans converts single coordinates, large files of coordinates, and live coordinates from a GPS. A large database is included, which contains thousands of coordinate systems — data and grids — from all over the world. The coordinate systems are categorized by country and can also be found by free text search. Custom datums and grids can be defined. Franson Technology Arkovagen 45, Johanneshov 121 55 Stockholm, Sweden 46-8-612-50-70 http://franson.com/coordtrans/ Parasoft has released C++Test 6.5, an automated unit testing and coding standard analysis toolsuite that prevents the occurrence of software errors as they develop. C++Test 6.5 finds and resolves software errors in C/C++ code with enhancements to both code analysis and unit testing as well as increased platform and team support. With Parasoft C++Test 6.5, source code can now be validated against over 600 C/C++ coding standards and guidelines that reflect industry standard best practices in code design and constructionaffecting reliability, performance, mainDr. Dobb’s Journal, December 2005
tainability, security, and compliance. A new RuleWizard Autocreator lets you easily create your own custom rules or guidelines by simply providing RuleWizard a sample of “bad code” as the guideline for automatic rule creation. Parasoft 101 East Huntington Drive Monrovia, CA 91016 626-256-3680 http://www.parasoft.com/ Quinn-Curtis has released Java versions of its QCChart2D Charting and QCRTGraph Real-Time Graphics software tools. The software supports linear, logarithmic, time/date, and polar coordinate systems. Data plot types include static and dynamic plotting of line plots, bar graphs, scatter plots, dials, gauges, meters, clocks, annunciators, and panel meter indicators. Users are able to interact with charts: adding annotations, selecting, marking and moving data points, displaying data tooltips, and zooming into a particular chart region. Quinn-Curtis 18 Hearthstone Drive Medfield, MA 02052 508-359-6639 http://www.quinn-curtis.com/ SoftIntegration has announced availability of ChExcel Toolkit 1.0. Ch is an embeddable C/C++ interpreter for crossplatform scripting, 2D/3D plotting, numerical computing, shell programming, and embedded scripting. ChExcel is a Microsoft Excel add-in. It embeds Ch (a C/C++ Interpreter) into Excel using Embedded Ch. ChExcel lets users access Ch statements, functions, programs, toolkits, and packages from Excel spreadsheets and Visual Basic for Applications (VBA). With ChExcel, Excel spreadsheets can be manipulated through C/C++ scripts. The data can be easily exchanged between Ch and Excel. SoftIntegration Inc. 216 F Street, #68 Davis, CA 95616 530-297-7398 http://www.softintegration.com/ DDJ Dr. Dobb’s Software Tools Newsletter What’s the fastest way of keeping up with new developer products and version updates? Dr. Dobb’s Software Tools e-mail newsletter, delivered once a month to your mailbox. This unique newsletter keeps you up-to-date on the latest in SDKs, libraries, components, compilers, and the like. To sign up now for this free service, go to http://www.ddj.com/maillists/.
87
SWAINE’S FLAMES
Dead Software Walking
I
’m sure some of you remember Randy Sutherland. Randy stopped by our place in September for a couple of days. On the first evening, we went out to eat; but on the second night, our restaurant was open, so Randy ordered the Epicurean Adventure. Very appropriate. Randy was my right-hand guy when I was editor-in-chief of DDJ back in the early 1980s. He was first managing editor, then editor. He went on to other jobs, including a long stint at Cisco, where he made a ton of money before getting out with — or for — his sanity. Randy’s always seemed eminently sane to me, but he’s a different kind of sane these days. I was getting e-mail diaries from him last year from remote locations that were always exotic and often downright dangerous-sounding, in North Africa and the Middle East, some of it reading like passages from The Sheltering Sky. So you can see why the Epicurean Adventure, or any adventure, would sound like his dish. Randy’s visit, coupled with the impending 30th anniversary of this magazine and other events of this year, put me in a nostalgic mood. Just the mood to mark the passage of a software company that was not just admired, but loved by many. Cyan has faded to black, gone white on white, been blue-penciled. It has fallen down the manhole, faded in the mist, seen the world beyond the mackerel, and has been riven from reality and exiled to the end of ages. It has been given an earth suit and sent to Headstone Park to curl up its toes and dance the horizontal tango. It was in 1987, my last year of working in an office, when brothers Rand and Robyn Miller wrote Manhole, the first CD-ROM-based game and a new kind of electronic entertainment. Manhole and every subsequent game from their company, Cyan, was a hit, and every one pushed the limits in the creation of virtual worlds. The titles include Cosmic Osmo and the Worlds Beyond the Mackerel, Spelunx, Myst, Riven, and various Myst sequels. The talented Miller brothers will continue to do creative work, probably in a variety of media, but their company ran out of money and came to an end in September of this year. With it disappears a gentle spirit that is lacking in most popular computer games today. Cyan will be missed. The Miller brothers wrote Manhole in HyperCard, and used HyperCard to prototype later games. Some would say that HyperCard has also Gone to the Place where Grampa Lives, but that case is more complicated. It’s true that Apple no longer sells or supports or acknowledges HyperCard, but software doesn’t really die until all the hardware it can run on dies. Apple’s working on that. HyperCard, also released in 1987, was a wonderful product, a development environment for ordinary users, almost a virtual computer on top of the Mac environment. It didn’t take a lot of time to learn the rudiments of HyperTalk, HyperCard’s scripting language, and start creating the miniapplications called “stacks.” Naive users could use stacks written by others, and a huge number of these quickly became available. HyperCard was a godsend to educators in particular. HyperCard had its limitations: It was object based, not object oriented, and it was originally black and white only. But it could have been brought up to date, made to work under OS X, and if that had happened it would still be popular today. In fact, a small but significant number of people still use it, on old Macs or under the Classic environment. Classic, however, will not run on the new Macintel machines. That would seem to spell the doom of HyperCard, but its fans are not giving up. The most obvious thing to do would be to create a HyperCard clone. It’s been done before: SuperCard, Plus, ToolBook, MetaCard. And Edinburgh-based Revolution (Mike Markkula is an investor) has built on the MetaCard engine to create a powerful multiplatform HyperCard-like product. But it’s different enough from HyperCard that some stackheads are still hoping somehow to get HyperCard running on Macintel. A recent baroque proposal involved using Basilisk, a 68K Mac emulator. It makes my head hurt.
Michael Swaine editor-at-large [email protected] 88
Dr. Dobb’s Journal, December 2005
http://www.ddj.com
