RELATIONAL MANAGEMENT and DISPLAY of SITE ENVIRONMENTAL DATA
RELATIONAL MANAGEMENT and DISPLAY of SITE ENVIRONMENTAL DATA David W. Rich, Ph.D.
LEWIS PUBLISHERS A CRC Press Company Boca Raton London New York Washington, D.C.
Library of Congress Cataloging-in-Publication Data Rich, David William, 1952Relational management and display of site environmental data / David W. Rich. p. cm. Includes bibliographical references and index. ISBN 1-56670-591-6 (alk. paper) 1. Pollution—Measurement—Data processing. 2. Environmental monitoring—Data processing. 3. Database management. I. Title. TD193 .R53 2002 628.5′028′ 7—dc21
2002019441
This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior permission in writing from the publisher. The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying. Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation, without intent to infringe.
Visit the CRC Press Web site at www.crcpress.com © 2002 by CRC Press LLC Lewis Publishers is an imprint of CRC Press LLC No claim to original U.S. Government works International Standard Book Number 1-56670-591-6 Library of Congress Card Number 2002019441 Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 Printed on acid-free paper
PREFACE
The environmental industry is changing, along with the way it manages data. Many projects are making a transition from investigation through remediation to ongoing monitoring. Data management is evolving from individual custom systems for each project to standardized, centralized databases, and many organizations are starting to realize the cost savings of this approach. The objective of Relational Management and Display of Site Environmental Data is to bring together in one place the information necessary to manage the data well, so everyone, from students to project managers, can learn how to benefit from better data management. This book has come from many sources. It started out as a set of course notes to help transfer knowledge about earth science computing and especially environmental data management to our clients as part of our software and consulting practice. While it is still used for that purpose, it has evolved into a synthesis of theory and a relation of experience in working with site environmental data. It is not intended to be the last word on the way things are or should be done, but rather to help people learn from the experience of others, and avoid mistakes whenever possible. The book has six main sections plus appendices. Part One provides an overview of the subject and some general concepts, including a discussion of system data content. Part Two covers system design and implementation, including database elements, user interface issues, and implementation and operation of the system. Part Three addresses gathering the data, starting with an overview of site investigation and remediation, progressing through gathering samples in the field, and ending with laboratory analysis. Part Four covers the data management process, including importing, editing, maintaining data quality, and managing multiple projects. Part Five is about using the data once it is in the database. It starts with selecting data, and then covers various aspects of data output and analysis including reporting and display; graphs; cross sections and similar displays; a large chapter on mapping and GIS; statistical analysis; and integration with other programs. Section Six discusses problems, benefits, and successes with implementing a site environmental data management system, along with an attempt to look into the future of data management and environmental projects. Appendices include examples of a needs assessment, a data model, a data transfer standard, typical constituent parameters, some exercises, a glossary, and a bibliography. A number of people have contributed directly and indirectly to this book, including my parents, Dr. Robert and Audrey Rich; Dr. William Fairley, my uncle and professor of geology at the University of Notre Dame; and Dr. Albert Carozzi, my advisor and friend at the University of Illinois. Numerous coworkers and friends at Texaco, Inc., Shell Oil Company, Sabine Corporation, Grant Environmental, and Geotech Computer Systems, Inc. helped bring me to the point professionally where I could write this book. These include Larry Ratliff, Jim Thomson, Dr. James L. Grant, Neil Geitner, Steve Wampler, Jim Quin, Cathryn Stewart, Bill Thoen, Judy Mitchell, Dr. Mike Wiley, and other Geotech staff members who helped with the book in various ways. Friends
in other organizations have also helped me greatly in this process, including Jim Reed of RockWare, Tom Bresnahan of Golden Software, and other early members of the Computer Oriented Geological Society. Thanks also go to Dr. William Ganus, Roy Widmann, Sherron Hendricks, and Frank Schultz of Kerr-McGee for their guidance. I would also like to specifically thank those who reviewed all or part of the book, including Cathryn Stewart (AquAeTer), Bill Thoen (GISNet), Mike Keester (Oklahoma State University), Bill Ganus and Roy Widmann (Kerr-McGee), Mike Wiley (The Consulting Operation), and Sue Stefanosky and Steve Clough (Roy. F. Weston). The improvements are theirs. The errors are still mine. Finally, my wife, business partner, and best friend, Toni Rich, has supported me throughout my career, hanging in there through the good times and bad, and has always done what she could to make our enterprise successful. She’s also a great proofreader. Throughout this book a number of trademarks and registered trademarks are used. The registered trademarks are registered in the United States, and may be registered in other countries. Any omissions are unintentional and will be remedied in later editions. Enviro Data and Spase are registered trademarks of Geotech Computer Systems, Incorporated. Microsoft, Office, Windows, NT, Access, SQL Server, Visual Basic, Excel, and FoxPro are trademarks or registered trademarks of Microsoft Corporation. Oracle is a registered trademark of Oracle Corporation. Paradox and dBase are registered trademarks of Borland International, Incorporated. IBM and DB2 are registered trademarks of International Business Machines Corporation. AutoCAD and AutoCAD Map are registered trademarks of Autodesk, Incorporated. ArcView is a registered trademark of Environmental Systems Research Institute, Incorporated. Norton Ghost is a trademark of Symantec Corporation. Apple and Macintosh are registered trademarks of Apple Computer, Incorporated. Sun is a registered trademark and Sparcstation is a trademark of Sun Microsystems. Capability Maturity Model and CMM are registered trademarks of The Software Engineering Institute of Carnegie Mellon University. Adobe and Acrobat are registered trademarks of Adobe Systems. Grapher is a trademark and Surfer is a registered trademark of Golden Software, Inc. RockWare is a registered trademark and RockWorks and Gridzo are trademarks of RockWare, Inc. Intergraph and GeoMedia are trademarks of Intergraph Corporation. Corel is a trademark and Corel Draw is a registered trademark of Corel Corporation. UNIX is a registered trademark of The Open Group. Linux is a trademark of Linus Torvalds. Use of these products is for illustration only, and does not signify endorsement by the author. A Web site has been established for updates, exercises, and other information related to this book. It is located at www.geotech.com/relman. I welcome your comments and questions. I can be reached by email at
[email protected]. David W. Rich
AUTHOR
David W. Rich is founder and president of Geotech Computer Systems, Inc. in Englewood, CO. Geotech provides off-the-shelf and custom software and consulting services for environmental data management, GIS, and other technical computing projects. Dr. Rich received his B.S. in Geology from the University of Notre Dame in 1974, and his M.S. and Ph.D. in Geology from the University of Illinois in 1977 and 1979, with his dissertation on “Porosity in Oolitic Limestones.” He worked for Texaco, Inc. in Tulsa, OK and Shell Oil Company in Houston, TX, exploring for oil and gas in Illinois and Oklahoma. He then moved to Sabine Corporation in Denver, CO as part of a team that successfully explored for oil in the Minnelusa Formation in the Powder River Basin of Wyoming. He directed the data management and graphics groups at Grant Environmental in Englewood, CO where he worked on several projects involving soil and groundwater contaminated with metals, organics, and radiologic constituents. His team created automated systems for mapping and cross section generation directly from a database. In 1986 he founded Geotech Computer Systems, Inc., where he has developed and supervised the development of custom and commercial software for data management, GIS, statistics, and Web data access. Environmental projects with which Dr. Rich has been directly involved include two Superfund wood treating sites, three radioactive material processing facilities, two hazardous waste disposal facilities, many municipal solid waste landfills, two petroleum refineries, and several mining and petroleum production and transportation projects. He has been the lead developer on three public health projects involving blood lead and related data, including detailed residential environmental measurements. In addition he has been involved in many projects outside of the environmental field, including a real-time Web-based weather mapping system, an agricultural GIS analysis tool, and database systems for petroleum exploration and production data, paleontological data, land ownership, health care tracking, parts inventory and invoice printing, and GPS data capture. Dr. Rich has been using computers since 1970, and has been applying them to earth science problems since 1975. He was a co-founder and president of the Computer Oriented Geological Society in the early 1980s, and has authored or co-authored more than a dozen technical papers, book chapters, and journal articles on environmental and petroleum data management, geology, and computer applications. He has taught many short courses on geological and environmental computing in several countries, and has given dozens of talks at various industry conventions and other events. When he is not working, Dr. Rich enjoys spending time with his family and riding his motorcycle in the mountains, and often both at the same time.
CONTENTS
PART ONE - OVERVIEW AND CONCEPTS ............................................................................1 CHAPTER 1 - OVERVIEW OF ENVIRONMENTAL DATA MANAGEMENT.............3 Concern for the environment...........................................................................................3 The computer revolution ..................................................................................................5 Convergence - Environmental data management ..........................................................7 Concept of data vs. information.......................................................................................8 EMS vs. EMIS vs. EDMS .................................................................................................8 CHAPTER 2 - SITE DATA MANAGEMENT CONCEPTS .............................................11 Purpose of data management .........................................................................................11 Types of data storage......................................................................................................12 Responsibility for data management .............................................................................18 Understanding the data ..................................................................................................19 CHAPTER 3 - RELATIONAL DATA MANAGEMENT THEORY................................21 What is relational data management?...........................................................................21 History of relational data management.........................................................................21 Data normalization .........................................................................................................22 Structured Query Language ..........................................................................................26 Benefits of normalization................................................................................................30 Automated normalization...............................................................................................31 CHAPTER 4 - DATA CONTENT ........................................................................................35 Data content overview ....................................................................................................35 Project technical data .....................................................................................................36 Project administrative data............................................................................................39 Project document data....................................................................................................41 Reference data.................................................................................................................42 Document management ..................................................................................................43 PART TWO - SYSTEM DESIGN AND IMPLEMENTATION ...............................................47 CHAPTER 5 - GENERAL DESIGN ISSUES......................................................................49 Database management software ....................................................................................49
Database location options...............................................................................................50 Distributed vs. centralized databases ............................................................................56 The data model................................................................................................................59 Data access requirements ...............................................................................................61 Government EDMS systems...........................................................................................63 Other issues .....................................................................................................................64 CHAPTER 6 - DATABASE ELEMENTS ...........................................................................69 Hardware and software components.............................................................................69 Units of data storage .......................................................................................................75 Databases and files..........................................................................................................76 Tables (“databases”).......................................................................................................76 Fields (columns)...............................................................................................................78 Records (rows).................................................................................................................79 Queries (views) ................................................................................................................79 Other database objects ...................................................................................................80 CHAPTER 7 - THE USER INTERFACE............................................................................85 General user interface issues..........................................................................................85 Conceptual guidelines .....................................................................................................86 Guidelines for specific elements .....................................................................................90 Documentation ................................................................................................................91 CHAPTER 8 - IMPLEMENTING THE DATABASE SYSTEM ......................................93 Designing the system.......................................................................................................93 Buy or build?...................................................................................................................97 Implementing the system ................................................................................................99 Managing the system ....................................................................................................103 CHAPTER 9 - ONGOING DATA MANAGEMENT ACTIVITIES...............................107 Managing the workflow................................................................................................107 Managing the data ........................................................................................................109 Administering the system .............................................................................................110 PART THREE - GATHERING ENVIRONMENTAL DATA................................................115 CHAPTER 10 - SITE INVESTIGATION AND REMEDIATION..................................117 Overview of environmental regulations ......................................................................117 The investigation and remediation process.................................................................119 Environmental Assessments and Environmental Impact Statements.......................121 CHAPTER 11 - GATHERING SAMPLES AND DATA IN THE FIELD ......................123 General sampling issues................................................................................................123 Soil..................................................................................................................................126 Sediment.........................................................................................................................127 Groundwater .................................................................................................................127 Surface water ................................................................................................................130 Decontamination of equipment ....................................................................................131 Shipping of samples ......................................................................................................131 Air...................................................................................................................................131
Other media...................................................................................................................132 Overview of parameters ...............................................................................................133 CHAPTER 12 - ENVIRONMENTAL LABORATORY ANALYSIS .............................139 Laboratory workflow ...................................................................................................139 Sample preparation.......................................................................................................140 Analytical methods........................................................................................................141 Other analysis issues .....................................................................................................145 PART FOUR - MAINTAINING THE DATA ..........................................................................149 CHAPTER 13 - IMPORTING DATA................................................................................151 Manual entry .................................................................................................................151 Electronic import ..........................................................................................................153 Tracking imports...........................................................................................................163 Undoing an import ........................................................................................................164 Tracking quality............................................................................................................165 CHAPTER 14 - EDITING DATA ......................................................................................167 Manual editing ..............................................................................................................167 Automated editing.........................................................................................................168 CHAPTER 15 - MAINTAINING AND TRACKING DATA QUALITY........................173 QA vs. QC......................................................................................................................173 The QAPP ......................................................................................................................173 QC samples and analyses..............................................................................................175 Data quality procedures ...............................................................................................181 Database support for data quality and usability ........................................................186 Precision vs. accuracy...................................................................................................187 Protection from loss ......................................................................................................188 CHAPTER 16 - DATA VERIFICATION AND VALIDATION......................................191 Types of data review .....................................................................................................191 Meaning of verification ................................................................................................191 Meaning of validation...................................................................................................193 The verification and validation process ......................................................................193 Verification and validation checks ..............................................................................194 Software assistance with verification and validation.................................................195 CHAPTER 17 - MANAGING MULTIPLE PROJECTS AND DATABASES...............199 One file or many?..........................................................................................................199 Sharing data elements...................................................................................................201 Moving between databases...........................................................................................201 Limiting site access........................................................................................................202 PART FIVE - USING THE DATA ............................................................................................203 CHAPTER 18 - DATA SELECTION.................................................................................205 Text-based queries ........................................................................................................205 Graphical selection........................................................................................................207 Query-by-form ..............................................................................................................210 CHAPTER 19 - REPORTING AND DISPLAY................................................................213
Text output ....................................................................................................................213 Formatted reports.........................................................................................................214 Formatting the result ....................................................................................................216 Interactive output .........................................................................................................223 Electronic distribution of data .....................................................................................224 CHAPTER 20 - GRAPHS ...................................................................................................225 Graph overview.............................................................................................................225 General concepts ...........................................................................................................226 Types of graphs .............................................................................................................227 Graph examples.............................................................................................................228 Curve fitting ..................................................................................................................232 Graph theory .................................................................................................................233 CHAPTER 21 - CROSS SECTIONS, FENCE DIAGRAMS, AND 3-D DISPLAYS.....235 Lithologic and wireline logs .........................................................................................235 Cross sections ................................................................................................................237 Profiles ...........................................................................................................................238 Fence diagrams and stick displays...............................................................................239 Block Diagrams and 3-D displays ................................................................................240 CHAPTER 22 - MAPPING AND GIS ...............................................................................243 Mapping concepts .........................................................................................................243 Mapping software .........................................................................................................251 Displaying data..............................................................................................................254 Contouring and modeling.............................................................................................256 Specialized displays.......................................................................................................262 CHAPTER 23 - STATISTICS AND ENVIRONMENTAL DATA..................................269 Statistical concepts ........................................................................................................269 Types of statistical analyses..........................................................................................273 Outliers and comparison with limits ...........................................................................275 Toxicology and risk assessment ...................................................................................277 CHAPTER 24 - INTEGRATION WITH OTHER PROGRAMS....................................279 Export-import ...............................................................................................................279 Digital output.................................................................................................................282 Export-import advantages and disadvantages ...........................................................282 Direct connection ..........................................................................................................283 Data warehousing and data mining.............................................................................285 Data integration ............................................................................................................286 PART SIX - PROBLEMS, BENEFITS, AND SUCCESSES...................................................287 CHAPTER 25 - AVOIDING PROBLEMS........................................................................289 Manage expectations.....................................................................................................289 Use the right tool ...........................................................................................................290 Prepare for problems with the data ............................................................................291 Plan project administration .........................................................................................292 Increasing the chance of a positive outcome...............................................................292
CHAPTER 26 - SUCCESS STORIES................................................................................293 Financial benefits ..........................................................................................................293 Technical benefits..........................................................................................................295 Subjective benefits ........................................................................................................296 CHAPTER 27 - THE FUTURE OF ENVIRONMENTAL DATA MANAGEMENT ...299 PART SEVEN - APPENDICES .................................................................................................301 APPENDIX A - NEEDS ASSESSMENT EXAMPLE.......................................................303 APPENDIX B - DATA MODEL EXAMPLE ....................................................................307 Introduction...................................................................................................................307 Conventions ...................................................................................................................307 Primary tables ...............................................................................................................308 Lookup tables ................................................................................................................312 Reference tables ............................................................................................................321 Utility tables...................................................................................................................324 APPENDIX C - DATA TRANSFER STANDARD ...........................................................327 Purpose ..........................................................................................................................327 Database background information ..............................................................................327 Data content ..................................................................................................................328 Acceptable file formats .................................................................................................332 Submittal requirements ................................................................................................334 Non-conforming data....................................................................................................335 APPENDIX D - THE PARAMETERS...............................................................................337 Overview........................................................................................................................337 Inorganic parameters ...................................................................................................338 Organic parameters ......................................................................................................340 Other parameters..........................................................................................................347 Method reference ..........................................................................................................348 APPENDIX E - EXERCISES..............................................................................................357 Database redesign exercise...........................................................................................357 Data normalization exercise .........................................................................................359 Group discussion - data management and your organization...................................360 Database redesign exercise solution ............................................................................360 Data normalization exercise solution...........................................................................361 Database software exercises .........................................................................................361 APPENDIX F - GLOSSARY ..............................................................................................363 APPENDIX G - BIBLIOGRAPHY ....................................................................................407 INDEX..........................................................................................................................................419
PART ONE - OVERVIEW AND CONCEPTS
CHAPTER 1 OVERVIEW OF ENVIRONMENTAL DATA MANAGEMENT
Concern for our environment has been on the rise for many years, and rightly so. At many industrial facilities and other locations toxic or potentially toxic materials have been released into the environment in large amounts. While the health impact of these releases has been quite variable and, in some cases, controversial, it clearly is important to understand the impact or potential impact of these releases on the public, as well as on the natural environment. This has led to increased study of the facilities and the areas around them, which has generated a large amount of data. More and more, people are looking to sophisticated database management technology, together with related technologies such as geographic information systems and statistical analysis packages, to make sense of this data. This chapter discusses this increasing concern for the environment, the growth of computer technology to support environmental data management, and then some general thoughts on environmental data management in an organization.
CONCERN FOR THE ENVIRONMENT The United States federal government has been regulating human impact on the environment for over a century. Section 13 of the River and Harbor Act of 1899 made it unlawful (with some exceptions) to put any refuse matter into navigable waters (Mackenthun, 1998, p. 20). Since then hundreds of additional laws have been enacted to protect the environment. This regulation occurs at all levels of government from international treaties, through federal and state governments, to individual municipalities. Often this situation of multiple regulatory oversight results in a maze of regulations that makes even legitimate efforts to improve the situation difficult, but it has definitely increased the effort to clean up the environment and keep it clean. Through the 1950s the general public had very little awareness or concern about environmental issues. In the 1960s concern for the environment began to grow, helped at least some by the book Silent Spring by Rachel Carson (Carson, 1962). The ongoing significance of this book is highlighted by the fact that a 1994 edition of the book has a foreword by then Vice President Al Gore. In this book Ms. Carson brought attention to the widespread and sometimes indiscriminate use of DDT and other chlorinated hydrocarbons, organic phosphates, arsenic, and other materials, and the impact of this use on ground and surface water, soil, plants, and animals. She cites examples of workers overcome by exposure to large doses of chemicals, and changes in animal populations after use of these chemicals, to build the case that widespread use of these materials is harmful. She also discusses the link between these chemicals and cancer.
4
Relational Management and Display of Site Environmental Data
Rachel Carson’s message about concern for the environment came at a time, the 1960s, when America was ready for a “back-to-the-earth” message. With the youth of America and others organizing to oppose the war in Vietnam, the two causes fit well together and encouraged each other’s growth. This was reflected in the music of the time, with many songs in the sixties and seventies discussing environmental issues, often combined with sentiments against the war and nuclear power. The war in Vietnam ended, but the environmental movement lives on. There are many examples of rock songs of the sixties and seventies discussing environmental issues. In 1968 the rock musical Hair warned about the health effects of sulfur dioxide and carbon monoxide. Zager and Evans in their 1969 song In The Year 2525 talked about taking from the earth and not giving back, and in 1970 the Temptations discussed air pollution and many other social issues in Ball of Confusion. Three Dog Night also warned about air pollution in their 1970 songs Cowboy and Out in the Country. Perhaps the best example of a song about the environment is Marvin Gaye’s 1971 song Mercy Mercy Me (The Ecology), in which he talked about oil polluting the ocean, mercury in fish, and radiation in the air and underground. In 1975 Joni Mitchell told farmers not to use DDT in her song Big Yellow Taxi, and the incomparable songwriter Bob Dylan got into the act with his 1976 song A Hard Rain’s A-gonna Fall, warning about poison in water and global hunger. It’s not a coincidence that this time frame overlaps all of the significant early environmental regulations. A good example of an organized environmental effort that started in those days and continues today is Earth Day. Organized by Senator Gaylord Nelson and patterned after teach-ins against the war in Vietnam, the first Earth Day was held on April 22, 1970, and an estimated 20 million people around the country attended, according to television anchor Walter Cronkite. In the 10 years after the first Earth Day, 28 significant pieces of federal environmental legislation were passed, along with the establishment of the U.S. Environmental Protection Agency (EPA) in December of 1970. The first major environmental act, the National Environmental Policy Act of 1969 (NEPA) predated Earth Day, and had the stated purposes (Yost, 1997) of establishing harmony between man and the environment; preventing or eliminating damage to the environment; stimulating the health and welfare of man; enriching the understanding of ecological systems; and establishment of the Council on Environmental Quality. Since that act, many laws protecting the environment have been passed at the national, state, and local levels. Evidence that public interest in environmental issues is still high can be found in the public reaction to the book A Civil Action (Harr, 1995). This book describes the experience of people in the town of Woburn, Massachusetts. A number of people in the town became ill and some died due to contamination of groundwater with TCE, an industrial solvent. This book made the New York Times bestseller list, and was later made into a movie starring John Travolta. More recently, the movie Erin Brockovich starring Julia Roberts covered a similar issue in California with Pacific Gas and Electric and problems with hexavalent chrome in groundwater causing serious health issues. Public interest in the environment is exemplified by the various watchdog organizations that track environmental issues in great detail. A good example of this is Scorecard.org, (Environmental Defense, 2001) a Web site that provides a very large amount of information on environmental current events, releases of toxic substances, environmental justice, and similar topics. For example, on this site you can find the largest releasers of pollutants near your residence. Sites like this definitely raise public awareness of environmental issues. It’s also important to point out that the environmental industry is big business. According to reports by the U.S. Department of Commerce and Environmental Business International (as quoted in Diener, Terkla, and Cooke, 2000), the environmental industry in the U.S. in 1998 had $188.7 billion in sales, up 1.6% from the previous year. It employed 1,354,100 people in 115,850 companies. The worldwide market for environmental goods and services for the same period was estimated to be $484 billion.
Overview of Environmental Data Management
5
Figure 1 - The author (front row center) examining state-of-the-art punch card technology in 1959
THE COMPUTER REVOLUTION In parallel with growing public concern for the environment has been growth of technology to support a better understanding of environmental conditions. While people have been using computing devices of some sort for over a thousand years and mainframe computers since the 1950s (see Environmental Computing History Timeline sidebar), the advent of personal computers in the 1980s made it possible to use them effectively on environmental projects. For more information on the history of computers, see Augarten (1984) and Evans (1981). Discussions of the history of geological use of computers are contained in Merriam (1983,1985). With the advent of Windows-based, consumer-oriented database management programs in the 1990s, the tools were in place to create an environmental data management system (EDMS) to store data for one or more facilities and use it to improve project management. Computers have assumed an increasingly important role in our lives, both at work and at home. The average American home contains more computers than bathtubs. From electronic watches to microwave ovens, we are using computers of one type or another a significant percentage of our waking hours. In the workplace, computers have changed from big number crunchers cloistered somewhere in a climate-controlled environment to something that sits on our desk (or our lap). No longer are computers used only for massive computing jobs which could not be done by hand, but they are now replacing the manual way of doing our daily work. This is as true in the earth science disciplines as anywhere else. Consequently, industry sages have suggested that those who do not have computer skills will be left behind in the next wave of automation of the
6
Relational Management and Display of Site Environmental Data
Environmental Computing History Timeline 1000 BC – The Abacus was invented (still in use). 1623 – The first mechanical calculator was invented by German professor Wilhelm Schickard. 1834 – Charles Babbage began work on the Analytical Engine, which was never completed. 1850 – Charles Lyell was the first person to use statistics in geology. 1876 – Alexander Graham Bell patented the telephone. 1890 – Herman Hollerith built the Tabulating Machine, which was the first successful mechanical calculating machine. 1899 – The River and Harbor Act was the first environmental law passed in the United States. 1943 – The Mark 1, an electromechanical calculator, was developed. 1946 – ENIAC (Electronic Numerator, Integrator, Analyzer and Computer) was completed. (Dick Tracy’s wrist radio also debuted in the comic strip.) 1947 – The transistor was invented by Bardeen, Brattain, and Shockley at Bell Labs. 1951 – UNIVAC, the first commercial computer, became available. 1952 – Digital plotters were introduced. 1958 – The integrated circuit was invented by Jack Kilby at Texas Instruments. 1962 – Rachel Carson’s Silent Spring is published, starting the environmental movement. 1965 – IBM white paper on computerized contouring appeared. 1969 – National Environmental Policy Act (NEPA) was enacted. 1970 – The first Earth Day was held. 1970 – Relational data management was described by Edwin Codd. 1971 – The first microprocessor, the Intel 4004, was introduced. 1973 – SQL was introduced by Boyce and Chamberlain. 1977 – The Apple II, the first widely accepted personal computer, was introduced. 1981 – IBM releases its Personal Computer. This was the computer that legitimized small computers for business use. 1984 – The Macintosh computer was introduced, the first significant use of a graphical user interface on a personal computer. 1985 – Windows 1.0 was released. 1990 – Microsoft shipped Windows 3.0, the first widely accepted version. 1994 – Netscape Navigator was released by Mosaic Communications, leading to widespread use of the World Wide Web. workplace. At the least, those who are computer aware will be in a better position to evaluate how computers can help them in their work. The growth that we have seen in computer processing power is related to Moore’s law (Moore, 1965; see also Schaller, 1996), which states that the capacity of semiconductor memory doubles every 18 months. The price-performance ratio of most computer components meets or exceeds this law over time. For example, I bought a 10 megabyte hard drive in 1984 for $800. In 2001 I could buy a 20 gigabyte hard drive for $200, a price-performance increase of 8000 times in 17 years. This averages to a doubling about every 16 months. Over the same time, PC processing speed has increased from 4 megahertz for $5000 to 1000 megahertz for $1000, an increase of 1250, a doubling every 20 months. These average to 18 months. So computers become twice as powerful every year and a half, obeying Moore’s law. Unlike 10 or especially 20 years ago, it is now usual in industrial and engineering companies for each employee to have a suitable computer on his or her desk, and for that computer to be networked to other people’s computers and often a server. This computing environment is a good base on which to build a data management system.
Overview of Environmental Data Management
7
As the hardware has developed, so has the data management software. It is now possible to outfit an organization with the software for a client-server data management system starting at $1,000 or $2,000 a seat. Users probably already have the hardware. Adding software customization, training, support, and other costs still allows a powerful data management system to be put in place for a cost which is acceptable for many projects. In general, computers perform best when problem solving calls for either deductive or inductive reasoning, and poorly when using subjective reasoning. For example, calculating a series of stratigraphic horizon elevations where the ground level elevation and the depth to the formation are known is an example of deductive reasoning. Computers perform optimally on problems requiring deductive reasoning because the answers are precise, requiring explicit computations. Estimating the volume of contamination or contouring a surface is an example of inductive reasoning. Inductive reasoning is less precise, and requires a skilled geoscientist to critique and modify the interpretation. Lastly, the feeling that carbonate aquifers may be more complex than clastic aquifers is an example of subjective reasoning. Subjective reasoning uses qualitative data and is the most difficult of all for computer analysis. In such instances, the analytical potential of the computer is secondary to its ability to store and graphically portray large amounts of information. Graphic capabilities are requisite to earth scientists in order to make qualitative data usable for interpretation. Another example of appropriate use of computers relative to types of reasoning is the distinction between verification and validation, which is discussed in detail in Chapter 16. Verification, which checks compliance of data with project requirements, is an example of deductive logic. Either a continuous calibration verification sample was run every ten samples or it wasn’t. Validation, on the other hand, which determines the suitability of the data for use, is very subjective, requiring an understanding of sampling conditions, analytical procedures, and the expected use of the data. Verification is easily done with computer software. How far software can go toward complete automation of the validation process remains to be seen.
CONVERGENCE - ENVIRONMENTAL DATA MANAGEMENT Efficient data management is taking on increased importance in many organizations, and yours is probably no exception. In the words of one author (Diamondstone, 1990, p. 3): Automated measuring equipment has provided rapidly increasing amounts of data. Now, the challenge before us is to assure sufficient data uniformity and compatibility and to implement data quality measures so that these data will be useful for integrative environmental problem solving. This is particularly true in organizations where many different types of site investigation and monitoring data are coming from a variety of different sources. Fortunately, software tools are now available which allow off-the-shelf programs to be used by people who are not computer experts to retrieve this data in a format that is meaningful to them. According to Finkelstein (1989, p. 3): Management is on the threshold of an explosive boom in the use of computers. A boom initiated by simplicity and ease of use. Managers and staff at all levels of an organization will be able to design and implement their own systems, thereby dramatically reducing their dependence on the data processing (DP) department, while still ensuring that DP maintains central control, so that application systems and their data can be used by others in the business. With the advent of relatively easy to use software tools such as Microsoft Windows and Microsoft Access, it is even more true now that individuals can have a much greater role in
8
Relational Management and Display of Site Environmental Data
satisfying their own data management needs. It is important to develop a data management approach that makes efficient use of these tools to solve the business problem of managing data within the organization. The environmental data management system that will result from implementation of a plan based on this approach will provide users with access to the organization’s environmental data to satisfy their business needs. It will allow them to expand their data retrievals as their needs change and as their skills develop. As with most business decisions, the decision to implement a data management system should be based on an analysis of the expected return on the time and money invested. In the case of an office automation project, some of the return is tangible and can be expressed in dollar savings, and some is intangible savings in efficiency in everyday operations. In general, the best approach for system implementation is to look for leverage points in the system where a great return can be had for a small cost. The question becomes: How do you improve the process to get the greatest savings? Often some examples of tangible returns can be identified within the organization. The benefits can best be seen from analyzing the impact of the data management system on the whole site investigation and remediation process. For example, during remediation you might be able, by more careful tracking and modeling of the contamination, to decrease the amount of waste to be removed or water to be processed. You may also be able to decrease the time required to complete the project and save many person-years of cost by making quality data available in a standardized format and in a timely fashion. For smaller sites, automating the data management process can provide savings by repetition. Once the system has been set up for one site and people trained to use it, that effort can be re-used on the next site. The intangible benefits of a data management system are difficult to quantify, but subjectively can include increased job satisfaction of project workers, a higher quality work product, and better decision making. The cumulative financial and intangible return on investment of these various benefits can easily justify reasonable expenditures for a data management system.
CONCEPT OF DATA VS. INFORMATION It is important to recognize that there is a difference between numbers and letters stored in a computer and useful information. Numbers stored in a computer, or printed out onto a sheet of paper, may not themselves be of any value. It is only when those numbers are presented in a form that is useful to the intended audience that they become useful information. The keys to making the transition from data to information are organization and access. It doesn't matter if you have a file of all the monitoring wells ever drilled; if you can't get the information you want out of the file, it is useless. Before any database is created, careful attention should be paid to how the data is going to be used, to ensure that the maximum use can be received from the effort. Statistics and graphics can be tremendously helpful in perceiving relationships among different variables contained in the data. As the power and ease-of-use of both general business programs and technical programs for statistics and graphics improves, it will become common to take a good look at the data as a set before working with individual members of the set. The next step is to move from information to knowledge. The difference between the two is understanding. Once you have processed the information and understand it, it becomes knowledge. This transition is a human activity, not a computer activity, but the computer can help by presenting the information in an understandable manner.
EMS VS. EMIS VS. EDMS A final overview issue to discuss is the relationship between EMS (environmental management systems), EMIS (environmental management information systems), and site EDMS (environmental data management systems). An EMS is a set of policies and procedures for managing
Overview of Environmental Data Management
9
Data is or Data are? Is “data” singular or plural? In this book the word data is used as a singular noun. Depending on your background, you may not like this. Many engineers and scientists think of data as the plural of “datum,” so they consider the word plural. Computer people view data as a chunk of stuff, and, like “chunk,” consider it singular. In one dictionary I consulted (Webster, 1984), data as the plural of datum was the third definition, with the first two being synonyms for “information,” which is always treated as singular. It also states that common usage at this time is singular rather than plural, and that “data can now be used as a singular form in English.” In Strunk and White (1935), a style manual that I use, the discussion of singular vs. plural nouns uses an example of the contents of a jar. If the jar contains marbles, its contents are plural. If it contains jam, its content is singular. You decide: Is data jam or marbles? environmental issues for an organization or a facility. An EMIS is a software system implemented to support the administration of the EMS (see Gilbert, 1999). EMIS usually has a focus on record keeping and reporting, and is implemented with the hope of improving business processes and practices. A site environmental data management systems (EDMS) is a software system for managing data regarding the environmental impact of current or former operations. EDMS overlaps partially with EMIS systems. For an operating facility, the EDMS is a part of the EMIS. For a facility no longer in operation, there may be no formal EMS or EMIS, but the EDMS is necessary to facilitate monitoring and cleanup.
CHAPTER 2 SITE DATA MANAGEMENT CONCEPTS
The size and complexity of environmental investigation and monitoring programs at industrial facilities continue to increase. Consequently the amount of environmental data, both at operating facilities and orphan sites, is growing as well. The volume of data often exceeds the capacity of simple tools like paper reports and spreadsheets. When that happens it is appropriate to implement a more powerful data management system and often the system of choice is a relational database manager. This section provides a top-down discussion of management of environmental data. It focuses on the purpose and goals of environmental data management, and on the types and locations of data storage. These issues should always be resolved before an electronic (or in fact any) data management system should be implemented.
PURPOSE OF DATA MANAGEMENT Why manage data electronically? Or why even manage it at all? Clear answers to these questions are critical before a successful system can be implemented. This section addresses some of the issues related to the purpose of data management. It all comes down to planning. If you understand the goal to be accomplished, you have a better chance of accomplishing it. There is only one real purpose of data management: to support the goals of the organization. These goals are discussed in detail in Chapter 8. No data management system should be built unless it satisfies one or more significant business or technical goals. Identification of these goals should be done prior to designing and implementing the system for two reasons. One reason is that the achievement of these goals provides the economic justification for the effort of building the system. The other reason is that the system is more likely to generate satisfactory results if those results are understood, at least to a first approximation, before the system is implemented and functionality is frozen. Different organizations have different things that make them tick. For some organizations, internal considerations such as cost and efficiency are most important. For others, outside appearances are equally or more important. The goals of the organization must be taken into consideration in the design of the system so that the greatest benefit can be achieved. Typical goals include: Improve efficiency – Environmental site investigation and remediation projects can involve an enormous amount of data. Computerized methods, if they are well designed and implemented,
12
Relational Management and Display of Site Environmental Data
Environmental problems are complex problems. Complex problems have simple, easy-tounderstand wrong answers. From Environmental Humor by Gerald Rich (1996), reprinted with permission can be a great help in improving the flow of data through the project. They can also be a great sink of time and effort if poorly managed. Maximize quality – Because of the great importance of the results derived from environmental investigation and remediation, it is critical that the quality of the output be maximized relative to the cost. This is not trivial, and careful data storage, and annotation of data with quality information, can be a great help in achieving data quality objectives. Minimize cost – No organization has an unlimited amount of money, and even those with a high level of commitment to environmental quality must spend their money wisely to receive the greatest return on their investment. This means that unnecessary costs, whether in time or money, must be minimized. Electronic data management can help contain costs by saving time and minimizing lost data. People tend to start working on a database without giving a lot of thought to what a database really is. It is more than an accumulation of numbers and letters. It is a special way to help us understand information. Here are some general thoughts about databases: A database is a model of reality – In many cases, the data that we have for a facility is the only representation that we have for conditions at that facility. This is especially true in the subsurface, and for chemical constituents that are not visible, either because of their physical condition or their location. The model helps us understand the reality – In general, conditions at sites are nearly infinitely complex. The total combination of geological, hydrological and engineering factors usually exceeds our ability to understand it without some simplification. Our model of the site, based on the data that we have, helps us to perform this simplification in a meaningful way. This understanding helps us make decisions – Our simplified understanding of the site allows us to make decisions about actions to be taken to improve the situation at the site. Our model lets us propose and test solutions based on the data that we have, identify additional data that we need, and then choose from the alternative solutions. The clearer the model, the better the decisions – Since our decisions are based on our databased model, it follows that we will make better decisions if we have a clear, accurate, up-to-date model. The purpose of a database management system for environmental data is to provide us the information to build accurate models and keep them current. Clearly information technology, including data management, is important to organizations. Linderholm (2001) reports the results of a study that asked business executives about the importance of information technology (IT) to their business. 70% reported that it was absolutely essential, and 20% said it was extremely valuable. The net increase in revenue attributable to IT, after accounting for IT costs, was estimated to be 20%, clearly a good return. 70% said that the role of IT in business strategy is increasing. In the environmental business the story must be similar, but perhaps not as strong. If you were to survey project managers today about the importance of data management on their projects, probably the percentage that said it was essential or extremely valuable would be less than the 90% quoted above, and maybe less than 50%. But as the amount of data for sites continues to grow, this number will surely increase.
TYPES OF DATA STORAGE Once the purpose of the system has been determined, the next step is to identify the data to be contained in the system and how it is to be stored. Some data must be stored electronically, while
Site Data Management Concepts
13
other data might not need to be stored this way. Implementers should first develop a thorough understanding of their existing data and storage methods, and then make decisions about how electronic storage can provide an improvement. This section will cover ways of storing site environmental data. The content of an EDMS will be discussed in Chapter 4.
Hard copy Since its inception, hard copy data storage has been the lifeblood of the environmental industry. Many organizations have thousands of boxes of paper related to their projects. The importance of this data varies greatly, but in many organizations, it is not well understood. A data management system for hard copy data is different from a data management system for digital data such as laboratory analytical results. The former is really a document management system, and many vendors offer software and other tools to build this type of system. The latter is more of a technical database issue, and can be addressed by in-house generated solutions or offthe-shelf or semi-custom solutions from environmental software vendors.
LAB REPORTS Laboratory analyses can generate a large volume of paper. Programs like the U.S.E.P.A. Contract Lab Program (CLP) specify deliverables that can be hundreds of pages for one sampling event. This paper is important as backup for the data, but these hundreds of pages can cause a storage and retrieval problem for many organizations. Often the usable data from the lab event, that is, the data actually used to make site decisions, may be only a small fraction of the paper, with the rest being quality assurance and other backup information.
DERIVED REPORTS Evaluation of the results of laboratory analysis and other investigation efforts usually results in a printed report. These reports contain a large amount of useful information, but over time can also become a storage and retrieval problem.
Electronic There are many different ways of organizing data for digital storage. There is no “right” or “wrong” way, but there are approaches that provide greater benefits than others in specific situations. People store environmental data a lot of different ways, both in database systems and in other file types. Here we will discuss two non-database ways of storing data, and several different database system designs for storing data.
TEXT FILES AND WORD PROCESSOR FILES The simplest way to manage environmental data is in text files. These files contain just the information of interest, with no formatting or information about the data structure or relationships between different data elements. Usually these files are encoded in ASCII, which stands for American Standard Code for Information Interchange and is pronounced like as′-kee. For this reason they are sometimes called ASCII files. Text files can be effectively used for storing and transferring small amounts of data. Because they lack “intelligence” they are not usually practical for large data sets. For example, in order to search for one piece of data in a text file you must look at every word until you find the one you are looking for, rather than using a more efficient method such as indexed searching used by data management programs. A variation on text files is word processor files, which contain some formatting and structure resulting from the word processing program that created them. An example of this would be the data in a table in a report. Again this works well only for small amounts of data.
14
Relational Management and Display of Site Environmental Data
SPREADSHEETS Over the years a large amount of environmental data has been managed in spreadsheets. This approach works for data sets that are small to medium in size, and where the display and retrieval requirements are relatively simple. For large data sets, a database manager program is usually required because spreadsheets have a limit to the number of rows and columns that they contain, and these limits can easily be exceeded by a large data set. For example, Lotus 123 has a limit of about 16,000 rows of data, and Excel 97 has a limit of 65,536 rows. Spreadsheets do have their place in working with environmental data. They are particularly useful for statistical analysis of data and for graphing in a variety of ways. Spreadsheets are for doing calculations. Database managers are for managing data. As long as both are used appropriately, the two together can be very powerful. The problem with spreadsheets occurs when they are used in situations where real data management is required. For example, it’s not unusual for organizations to manage quarterly groundwater monitoring data using spreadsheets. They can do statistics on the data and print reports. Where the problem becomes evident is when it becomes necessary to do a historical analysis of the data. It can be very difficult to tie the data together. The format of the spreadsheets may have evolved over time. The file for one quarter may be missing or corrupted. Suddenly it becomes a chore to pull all of the data together to answer a question such as “What is the trend of the sulfate values over the last five years?”
DATABASE MANAGERS For storing large amounts of data, and where immediate calculations are not as important, database managers usually do a better job than spreadsheets, although the capabilities of spreadsheets and databases certainly overlap somewhat. The better database managers allow you to store related data in several different tables and to link them together based on the contents of the data. Many database manager programs have a reputation for not being very easy to use, partly because of the sheer number of options available. This has been improved with the menu-driven interfaces that are now available. These interfaces help with the learning curve, but data management software, especially database server software, can still be very difficult to master. Many database manager programs provide a programming language, which allows you to automate tasks that you perform often or repeatedly. It also allows you to configure the system for other users. This language provides the tools to develop sophisticated applications programs for nearly any data handling need, and provides the basis for some commercial EDMS software. Database managers are usually classified by how they store and relate data. The most common types are flat files, hierarchical, network, object-oriented, and relational. Most use the terminology of “record” for each object in the database (such as a well or sample location) and “field” for each type of information on each object (such as county or collection date). For information on database management concepts see Date (1981) and Rumble and Hampel (1984). Sullivan (2001) quotes a study by the University of California at Berkeley that humans have generated 12 exabytes (an exabyte is over 1 million terabytes, or a million trillion bytes) of data since the start of time, and will double this in the next two and a half years. Currently, about 20% of the world’s data is contained in relational databases, while the rest is in flat files, audio, video, pre-relational, and unstructured formats.
Flat file A flat file is a two-dimensional array of data organized in rows and columns similar to a spreadsheet. This is the simplest type of database manager. All of the data for a particular type of object is stored in a single file or table, and each record can have one instance of data for each field. A good analogy is a 3"×5" card file, where there is one card (record) for each item being tracked in the database, and one line (field) for each type of information stored.
Site Data Management Concepts
15
Flat file database managers are usually the cheapest to buy, and often the easiest to use, but the complexity of real-world data often requires more power than they can provide. In a flat file storage system, each row represents one observation, such as a boring or a sample. Each column contains the same kind of data. An example of a flat file of environmental data is shown in the following table: Well B-1 B-1 B-2 B-2 B-2 B-3 B-3
Elev 725 725 706 706 706 714 714
X 1050 1050 342 342 342 785 785
Y 681 681 880 880 880 1101 1101
SampDate 2/3/96 5/8/96 11/4/95 2/3/96 5/8/96 2/3/96 5/8/96
Sampler JLG DWR JAM JLG DWR JLG CRS
As .05 .05 3.7 2.1 1.4 .05 .05
AsFlag not det not det detected detected detected not det not det
Cl
ClFlag
.05 9.1 8.4 7.2
not det detected detected detected
.05
not det
pH 6.8 6.7 5.2 5.3 5.8 8.1 7.9
Figure 2 - Flat file of environmental data
In this table, each line is the result of one sampling event for an observation well. Since the wells were sampled more than once, and analyzed for multiple parameters, information specific to the well, such as the elevation and location (X and Y), is repeated. This wastes space and increases the chance for error since the same data element must be entered more than once. The same is true for sampling events, represented here by the date and the initials of the person doing the sampling. Also, since the format for the analysis results requires space for each value, if the value is missing, as it is for some of the chloride measurements, the space for that data is wasted. In general, flat files work acceptably for managing small amounts of data such as individual sampling events. They become less efficient as the size of the database grows. Examples of flat file data management programs are FileMaker Pro (www.filemaker.com) and Web-based database programs such as QuickBase (www.quickbase.com).
Hierarchical In the hierarchical design, the one-to-many relationship common to many data sets is formalized into the database design. This design works well for situations such as multiple samples for each boring, but has difficulty with other situations such as many-to-many relationships. This type of program is less common than flat files or relational database managers, but is appropriate for some types of data. In a hierarchical database, data elements can be viewed as branches of an inverted tree. A good example of a hierarchical database might be a database of organisms. At the top would be the kingdom, and underneath that would be the phyla for each kingdom. Each phylum belongs to only one kingdom, but each kingdom can have several phyla. The same continues down the line for class, order, and so on. The most important factor in fitting data into this scheme is that there must be no data element at one level that needs to be under more than one element at a higher level. If a crinoid could be both a plant and an animal at the same time, it could not be classified in a hierarchical database by phylogeny (which biological kingdom it evolved from). Environmental site data is for the most part hierarchical in nature. Each site can have many monitoring wells. Each well can have many samples, either over time or by depth. Then each sample can be analyzed for multiple constituents. Each constituent analysis comes from one specific sample, which comes from one well, which comes from one site. A data set which is inherently hierarchical can be stored in a relational database manager, and relational database managers are somewhat more flexible, so pure hierarchical database managers are now rare.
16
Relational Management and Display of Site Environmental Data
Network In the network data model, multiple relationships between different elements at the same level are easy to manage. Hypertext systems (such as the World Wide Web) are examples of managing data this way. Network database managers are not common, but are appropriate in some cases, especially those in which the interrelationships among data are complex. An example of a network database would be a database of authors and articles. Each author may have written many articles, and each article may have one or more authors. This is called a “many-to-many” relationship. This is a good project for a network database manager. Each author is entered, as is each article. Then the links between authors and articles are established. The data elements are entered, and then the network established. Then an article can be called up, and the information on its authors can be retrieved. Likewise, an author can be named, and his or her articles listed. A network data topology (geometric configuration) can be stored in a relational database manager. A “join table” is needed to handle the many-to-many relationships. Storing the above article database in a relational system would require three tables, one for authors, one for articles, and a join table with the connections between them.
Object oriented This relatively recent invention stores each data element as an object with properties and methods encapsulated (wrapped up) into each object. This is a deviation from the usual separation of code and data, but is being used successfully in many settings. Current object-oriented systems do not provide the data retrieval speed on large data sets provided by relational systems. Using this type of software involves a complete re-education of the user, since different terminology and concepts are used. It is a very powerful way to manipulate data for many purposes, and is likely to see more widespread use. Some of the features of object-oriented databases are described in the next few paragraphs. Encapsulation – Traditional programming languages focus on what is to be done. This is referred to as “procedural programming.” Object-oriented programming focuses on objects, which are a blend of data and program code (Watterson, 1989). In a procedural paradigm (a paradigm is an approach or model), the data and the programs are separate. In an object-oriented paradigm, the objects consist of data that knows what to do with itself, that is, objects contain methods for performing actions. This is called encapsulation. Thus, instead of applying procedures to passive data, in object-oriented programming systems (OOPS), methods are part of the objects. Some examples of the difference between procedural systems and OOPS might be helpful. In a procedural system, the data for a well could contain a field for well type, such as monitoring well or soil boring. The program operating on the data would know what symbol to draw on the map based on the contents of that field. In an OOPS the object called “soil boring” would include a method to draw its symbol, based on the data content (properties) of the object. Properties of objects in OOPS are usually loosely typed, which means that the distinction between data types such as integers and characters is not rigorously defined. This can be useful when, as is often the case, a numeric property such as depth to a particular formation needs to be filled with character values such as NP (not present) or NDE (not deep enough). For another illustration, imagine modeling a rock or soil body subject to chemical and physical processes such as leaching or neutralization using an OOPS. Each mineral phase would be an object of class “mineral,” while each fluid phase would be an object of class “fluid.” Methods known to the objects would include precipitation, dissolution, compaction, and so on. The model is given an initial condition, and then the objects interact via messages triggering methods until some final state is reached. Inheritance – Objects in an OOPS belong to classes, and members of a particular class share the same methods. Also, similar classes of objects can inherit properties and methods from an existing class. This feature, called inheritance, allows a building-block approach to designing a
Site Data Management Concepts
17
database system by first creating simple objects and then building on and combining them into more complex objects. In this way, an object called “site” made up of “well” objects would know how to display itself with no additional programming. Message Passing – An object-oriented program communicates with objects via messages, and objects can exchange messages as well. For example, an object of class “excavated material” could send a message to an object of class “remediation pit” which would update the property “remaining material” within object “remediation pit.” Polymorphism – A method is part of an object, and is distinct from messages between objects. The objects “well” and “boring” could both contain the method “draw yourself,” and sending the “draw yourself” message to one or the other object will cause a similar but different result. This is referred to as polymorphism. Object-oriented programming directly models the application, with messages being passed between objects being the analog of real-world processes (Thomas, 1989). Software written in this way is easier to maintain because programmers, other than the author, can easily comprehend the program code. Since program code is easily reusable, development of complex applications can be done more quickly and smoothly. Encapsulation, message passing, inheritance, and polymorphism give OOPS developers very different tools from those provided by traditional programming languages. Also, OOPS often use a graphical user interface and large amounts of memory, making them more suitable to high-end computer systems. For these reasons, OOPS have been slow in gaining acceptance, but they are gaining momentum and are considered by many to be the programming system of the future. Examples of object-oriented programming languages include Smalltalk developed by Xerox at the Palo Alto Research Center in the 1970s (Goldberg and Robson, 1983); C++, which is a superset of the C programming language (Soustrup, 1986); and Hypercard for the Macintosh. NextStep, the programming environment for the Next computer, also uses the object-oriented paradigm. There are several database management programs that are designed to be object oriented, which means that their primary data storage design is to store objects. Also, a number of relational database management systems have recently added object data types to allow object-oriented applications to use them as data repositories, and are referred to as Object-Relational systems.
Relational Relational database managers and SQL are discussed in much greater detail in Chapter 3, and are described here briefly for comparison with other database manager types. In the relational model, data is stored in one or more tables, and these tables are related, that is, they can be joined together, based on data elements within the tables. This allows storage of data where there may be many pieces of one type of information related to one object (one-to-many relationship), as well as other relationships such as hierarchical and many-to-many. In many cases, this has been found to be the most efficient form of data storage for large, complicated databases, because it provides efficient data storage combined with flexible data retrieval. Currently the most popular type of database manager program is the relational type. A file of monitoring well data provides a good example of how real-world data can be stored in a relational database manager. One table is created which contains the header data for the well including location, date drilled, elevation, and so on, with one record for each well. For each well, the driller or logger will report a different number of formation tops, so a table of formation tops is created, with one record for each top. A unique identifier such as well ID number relates the two tables to each other. Each well can also have one or more screened intervals, and a table is created for each of those data types, and related by the same ID number. Each screened interval can have multiple sampling events, with a description for each, so another table can be created for these sample events, which can be related by well ID number and sample event number. Very complex
18
Relational Management and Display of Site Environmental Data
systems can be created this way, but it often will take a program, written in the database language, to keep track of all the relationships and allow data entry, updating, and reporting. The most popular way of interacting with relational database managers is Structured Query Language (SQL, sometimes incorrectly pronounced “sequel,” see below). SQL provides a powerful, flexible way of retrieving, adding and changing data in a relational system. A typical SQL query might look like this: SELECT X_COORD, Y_COORD, COLLAR_ELEV - MUDDY_TOP, SYMBOL_CODE FROM WELLS, TOPS WHERE MUDDY_TOP > 100 AND WELLS.WELL_ID = TOPS.WELL_ID This query would produce a list where the first column is the X coordinate for a well, the second column is the Y coordinate, the third column is the difference between the well elevation and the depth to the top of the Muddy Formation, and the fourth column is the symbol code. Only the wells for which the Muddy Formation is deeper than 100 would be listed. The X and Y coordinates, the elevation, and the symbol code come from the WELLS table and the Muddy Formation top comes from the TOPS table. The last line is the “relational join” that hooks the two tables together for this query, based on WELL_ID, which is a field common to both tables. A field that relates two tables like this is called a “key.” The output from this query could be sent to a mapping program to make a contour map of the Muddy structure. Most relational database managers have SQL as their native retrieval language. The rest usually have an add-in capability to handle SQL queries.
XML XML (eXtensible Markup Language) was developed as a data transfer format, and has become increasingly popular for exchanging data on the Internet, especially in business-to-business transactions. The use of XML for transferring data is discussed in Chapter 24. Database management products are now starting to appear that use XML as their data storage format. (For example, see Dragan, 2001.) As of this writing these products are very immature. One example product costs $50,000 and does not include a report writer module. This is expected with a new product category. What is not clear is whether this data storage approach will catch on and replace more traditional methods, especially relational data management systems. This may be unlikely, especially since relational software vendors are adding the capability to store XML data in their programs, which are much more mature and accepted. Given that XML was intended as a transfer format, it’s not clear that it is a good choice for a storage format. It will be interesting to see if database products with XML as their primary data storage type become popular.
RESPONSIBILITY FOR DATA MANAGEMENT A significant issue in many organizations is who is responsible for data management. In some organizations, data management is centralized in a data management group. In others, the project team members perform the data management. Some organizations outsource data management to a consultant. Finally, some groups use a combination of these approaches. Each of these options will be discussed briefly, along with its pros and cons. Dedicated data management group – The thinking in this approach is that the organization can have a group of specialists who manage data for all of the projects. This is often done in conjunction with the group (which may be the same people) that performs validation on the data. The advantages of this are that the people develop specialized skills and expertise that allows them to manage the data efficiently. They can respond to requests for custom processing and output, because they have mastered the tools that they use to manage data. The disadvantage is that they
Site Data Management Concepts
19
may not have hands-on knowledge of the project and its data, which may be necessary to recognize and remedy problems. They need to be kept in the loop, such as when a new well is drilled, or when the laboratory is asked to analyze for a different suite of constituents, so that they can react appropriately. Data management by the project team – Here the focus is on the benefit of project knowledge rather than data management expertise. The people managing the data are likely to be involved in decisions that affect the data, and should be in a position to respond to changes in the data being gathered and reported. They might have problems, though, when something is asked of them that is beyond their data management expertise, because that is only part of what they are expected to do. Outsourcing to a consultant – A very common practice is to outsource the data management to a consultant, often the one that is gathering the data. Then the consultant has to decide between one of the previous approaches. This may be the best option when staff time or expertise is not available in-house to do the data management. The price can be loss of control over the process. A team effort – Sometimes the best solution is a team effort. In this scenario, project team members are primarily responsible for the data management process. They are backed up by data management specialists, either in-house or through a consultant, to help them when needs change or problems occur. Project staff may outsource large data gathering, data entry, or cleanup projects, especially when it is necessary to “catch up,” for example, to bring a lot of historical data from disparate sources into a comprehensive database to satisfy some specific project requirements. The team approach has the potential to be the strongest, because it allows the project team to leverage the strengths of the different available groups. It does require good management skills to keep everyone on the same page, especially as deadlines approach.
UNDERSTANDING THE DATA It is extremely important that project managers, data administrators, and users of the software have a complete understanding of the data in the database and how it is to be used. It is important for them to understand the data structure. It is even more important for them to understand the content. The understanding of the structure refers to how data elements from the real world are placed in fields in the database. Many DBMS programs allow comments to be associated with fields, either in the data model, or on editing forms, or both. These comments can assist the user with understanding how the data is to be entered. Additional information on data structure is contained in Chapter 4. Once you know where different data elements are to go, you must also know what data elements in the database mean. This is true of both primary data elements (data in the main tables) and coded values (stored in lookup tables). The content can be easily misunderstood. A number of issues regarding data content are discussed in Parts Three and Four.
CHAPTER 3 RELATIONAL DATA MANAGEMENT THEORY
The people using an EDMS will often come to the system with little or no formal training or experience in environmental data management. In order to provide a conceptual framework on which they can build their expertise in using the system, this section provides an overview of the fundamentals of relational management of environmental data. This section starts with a discussion of the meaning and history of relational data management. This is followed by a description of breaking data apart with data normalization, and using SQL to put it back together again.
WHAT IS RELATIONAL DATA MANAGEMENT? Relational data management means organizing the data based on relationships between items within the database. It involves designing the database so that like data elements are grouped into tables together. This process is called data normalization. Then the data can be joined back together for retrievals, usually using the SQL data retrieval language. The key elements of relational data storage are: • • •
Tables – Database object containing records and fields where the data is stored. Examples: Samples, Analyses. Fields – Data elements (columns) within the table. Examples: Parameter Name, Value. Records – Items being stored (rows) within the table. Example: Arsenic for last quarter.
Each of these items will be discussed in much greater detail later.
HISTORY OF RELATIONAL DATA MANAGEMENT Prior to 1970, data was primarily managed in hierarchical and networked storage systems. Edwin Codd, an IBM Fellow working at the San Jose research lab, became concerned about protecting users from needing to understand the internal representation of the data, and published a paper entitled “A Relational Model of Data for Large Shared Data Banks” (Codd, 1970). This set off a debate in industry about which model was the best. IBM developed System/R and a team at Berkeley developed INGRES, both prototype relational data management systems built in the mid1970s.
22
Relational Management and Display of Site Environmental Data
SQL was the database language for System/R, and was first described in Boyce and Chamberlain (1973). SQL is sometimes pronounced “sequel.” This is not really correct. In the early 1970s Edwin Codd and others in IBM’s research center were working on relational data management, and Donald Chamberlain and others developed a language to work with relational databases (Gagnon, 1998). This language was called Structured English Query Language, or SEQUEL. This was later revised and renamed SEQUEL/2. SQL as it is currently implemented is not the same as these early languages. And IBM stopped using the name SEQUEL for legal reasons. So it is not correct to call SQL “sequel.” It’s better to just say the letters, but many call it “sequel” anyway. In 1977, a contract programming company called Software Development Laboratories (SDL) developed a database system for the Central Intelligence Agency for a project called “Oracle.” The company released the program, based on System/R and SQL, in 1979. This became the first commercial relational database management system, and ran on IBM mainframes and Digital VAX and UNIX minicomputers. SDL assumed the name Oracle for both the company and the product. IBM followed in 1981 with the release of SQL/DS, and with DB2 in 1983 (Finkelstein, 1989). Oracle and DB2 are still active products in widespread use. The American National Standards Institute (ANSI) accepted SQL as a standardized fourth generation (4GL) language in 1986, with several revisions since then.
DATA NORMALIZATION Often the process of storing complex types of data in a relational database manager is improved by a technique called data normalization. Usually the best database designs have undergone this process, and are referred as a “normalized data model.” The normalization process separates data elements into a logical grouping of tables and fields.
Definition Normalization of a database is a process of grouping the data elements into tables in a way that reduces needless duplication and wasted space while allowing maximum flexibility of data retrieval. The concepts of data normalization were developed by Edwin Codd, and the first three steps of normalization were described in his 1972 paper (Codd, 1972). These steps were expanded and developed by Codd and others over the next few years. A good summary of this work is contained in Date (1981). Figure 3 shows an example of a simple normalized data model for site environmental data. This figure is known as an entity-relationship diagram, or E-R diagram. In Figure 3, and in other E-R diagrams used in this book, each box represents a table, with the name of the table shown at the top. The other words in each box represent the fields in that table. These are the “entities.” The lines between the boxes represent the relationships between the tables, and the fields used for the relationships. All of the relationships shown are “one-to-many,” signified by the number one and the infinity symbol on the ends of the join lines. That means that one record in one table can be related to many records in the other table.
The Five Normal Forms Most data analysts recognize five levels of normalization. These forms, called First through Fifth Normal Form, represent increasing levels of organization of the data. Each level will be discussed briefly here, with examples of a simple environmental data set organized into each form.
Relational Data Management Theory
23
Figure 3 - Simple normalized data model for site environmental data
Tools are now available to analyze a data set and assist with normalizing it. The Access Table Analyzer Wizard (see Figure 14, later) is one example. These tools usually require that the data be in First Normal Form (rows and columns, no repeating groups) before they can work with it. In this section we will go through the process of normalizing a data set. We will start with a flat file of site analytical data in a format similar to the way it would be received from a laboratory. This is shown in Figure 4. Well B-1 B-1 B-2 B-2 B-2 B-3 B-3
Elev 725 725 706 706 706 714 714
X 1050 1050 342 342 342 785 785
Y 681 681 880 880 880 1101 1101
SampDate 2/3/96 5/8/96 11/4/95 2/3/96 5/8/96 2/3/96 5/8/96
Sampler JLG DWR JAM JLG DWR JLG CRS
As .05 .05 3.7 2.1 1.4 .05 .05
AsFlag not det not det detected detected detected not det not det
Cl
ClFlag
.05 9.1 8.4 7.2
not det detected detected detected
.05
not det
pH 6.8 6.7 5.2 5.3 5.8 8.1 7.9
Figure 4 - Environmental data prior to normalization - Problems: Repeating Groups, Redundancy
First Normal Form – First we will convert our flat file to First Normal Form. In this form, data is represented as a two-dimensional array of data, like a flat file. Unlike some flat files, a first normal form table has no repeating groups of fields. In the flat file illustration in Figure 4, there are columns for arsenic (As), chloride (Cl), and pH, and detection flags for arsenic and chloride. This design requires that space be allocated for every constituent for every sample, even though some constituents were not measured. Converting this table to first Normal Form Results in the configuration shown in Figure 5.
24
Well B-1 B-1 B-1 B-1 B-1 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-3 B-3 B-3 B-3 B-3
Relational Management and Display of Site Environmental Data
Elev 725 725 725 725 725 706 706 706 706 706 706 706 706 706 714 714 714 714 714
X 1050 1050 1050 1050 1050 342 342 342 342 342 342 342 342 342 785 785 785 785 785
Y 681 681 681 681 681 880 880 880 880 880 880 880 880 880 1101 1101 1101 1101 1101
SampDate 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96 11/4/95 11/4/95 11/4/95 2/3/96 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96
Sampler JLG JLG DWR DWR DWR JAM JAM JAM JLG JLG JLG DWR DWR DWR JLG JLG CRS CRS CRS
Param As pH As Cl pH As Cl pH As Cl pH As Cl pH As pH As Cl pH
Value .05 6.8 .05 .05 6.7 3.7 9.1 5.2 2.1 8.4 5.3 1.4 7.2 5.8 .05 8.1 .05 .05 7.9
Flag not det not det not det detected detected detected detected detected detected not det not det not det
Figure 5 - Environmental data in first normal form (no repeating groups) - Problem: Redundancy
In this form, there is a line for each constituent that was measured for each well. There is no wasted space for the chloride measurements for B-1 or B-3 for 2/3/96. There is, however, quite a bit of redundancy. The elevation and location are repeated for each well, and the sampler’s initials are repeated for each sample. This redundancy is addressed in Second Normal Form. Second Normal Form – In this form, redundant data is moved to separate tables. In the more formal terminology of data modeling, data in non-key columns must be fully dependent on the primary key for the table. Key columns are the columns that uniquely identify a record. In our example, the data that uniquely identifies each row in the analytical table is a combination of the well, the sampling date, and the parameter. The above table has data, such as the elevation and sample date, which is not dependent on the entire compound key, but on only part of the key. Elevation depends on well but not on sample date, and sampler depends on well and sample date but not on parameter. In order to convert our table to Second Normal Form, we must separate it into three tables as shown in Figure 6. Third Normal Form – This form requires that the table conform to the rules for First and Second Normal Form, and that all non-key columns of a table be dependent on the table’s primary key and independent of one another. Once our example data set has been modified to fit Second Normal Form, it also meets the criteria for Third Normal Form, since all of the non-key values are dependent on the key fields and not on each other. Fourth Normal Form – The rule for Fourth Normal Form is that independent data entities cannot be stored in the same table where many-to-many relationships exist between these entities. Many-to-many relationships cannot be expressed as simple relationships between entities, but require another table to express this relationship. Our tables as described above meet the criteria for Fourth Normal Form.
Relational Data Management Theory
Stations Well B-1 B-2 B-3
Elev 725 706 714
Analyses X 1050 342 785
Y 681 880 1101
Samples Well B-1 B-1 B-2 B-2 B-2 B-3 B-3
25
SampDate 2/3/96 5/8/96 11/4/95 2/3/96 5/8/96 2/3/96 5/8/96
Sampler JLG DWR JAM JLG DWR JLG CRS
Well B-1 B-1 B-1 B-1 B-1 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-3 B-3 B-3 B-3 B-3
SampDate 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96 11/4/95 11/4/95 11/4/95 2/3/96 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96
Param As pH As Cl pH As Cl pH As Cl pH As Cl pH As pH As Cl pH
Value .05 6.8 .05 .05 6.7 3.7 9.1 5.2 2.1 8.4 5.3 1.4 7.2 5.8 .05 8.1 .05 .05 7.9
Flag not det not det not det detected detected detected detected detected detected not det not det not det
Figure 6 - Environmental data in second normal form - Problems: Compound Keys, Repeated Values
Fifth Normal Form – Fifth Normal Form requires that you be able to recreate exactly the original table from the tables into which it was decomposed. Our tables do not meet this criterion, since we cannot duplicate the repeating groups (value and flag for As, Cl, and pH) from our original table because we don’t know the order in which to display them. In order to overcome this, we need to add a table with the display order for each parameter. Param 1 2 3
Parameter As Cl pH
Order 1 2 3
Figure 7 -Parameters table for fifth normal form
The Flag field can be handled in a similar way. Note that when we handle the flag as a lookup table (a table containing values and codes), we are required to have values for that field in all of the records, or else records will be lost in a join at retrieval time. We have used “v” for detected value for the pH results, but a different flag could be used if desired to signify that these are different from the chemical values. Removal of compound keys – In order to minimize redundant data storage and to decrease the opportunity for error, it is often useful to remove compound keys from the tables and replace them with an artificial key that represents the removed values. For example, the Analyses table has a compound key of Well and SampDate. This key can be replaced by a SampleID field, which points to the sample table. Likewise, the parameter name can be removed from the Analyses table and a code inserted in its place. These keys can be AutoNum fields maintained by the system, and
26
Relational Management and Display of Site Environmental Data
don’t have to have any meaning relative to the data other than that the data is assigned to that key. Keys of this type are called synthetic keys and, because they are not compound, are also called simple keys. In many data management systems, some activities are made much easier by simple synthetic keys. For example, joining records between tables, and especially more complicated operations like pivot tables, can be much easier when there is a single field to use for the operation. The final result of the conversion to Fifth Normal Form is shown in Figure 8. The SampleID field in the Samples table is called a primary key, and must be unique for each record in the table. The SampleID field in the Analyses table is called a foreign key because it is the primary key in a different (foreign) but related table.
STRUCTURED QUERY LANGUAGE Normalization of a data model usually breaks the data out into multiple tables. Often to view the data in a way that is meaningful, it is necessary to put the data back together again. You can view this as “de-normalizing” or reconstituting the data. The tool most commonly used to do this is Structured Query Language, or SQL. The following sections provide an overview of SQL, along with examples of how it is used to provide useful data.
Overview of SQL The relational data model provides a way to store complex data in a set of relatively simple tables. These tables are then joined by key fields, which are present in more than one table and allow the data in the tables to be related to each other based on the values of these keys. Structured Query Language (SQL) is an industry-standard way of retrieving data from a relational database management system (RDBMS). There are a number of good books available on the basics of SQL, including van der Lans (1988) and Trimble and Chappell (1989).
How SQL is used In the simplest sense, an SQL query can take the data from the various tables in the relational model and reconstruct them into a flat file again. The benefit is that the format and content of the resulting grid of data can be different each time a retrieval is performed, and the format of the output is somewhat independent of the structure of the underlying tables. In other words, the presentation is separate from the contents. There are two parts to SQL: the data definition language (DDL) for creating and modifying the structure of tables, and the data manipulation language (DML) for working with the data itself. Access and other graphical data managers replace the SQL DDL with a graphical system for creating and editing table structures. This section will discuss SQL DML, which is used for inserting, changing, deleting, and retrieving data from one or more relational tables. The SQL keywords for changing data are INSERT, UPDATE, and DELETE. As you would expect, INSERT places records in a table, UPDATE changes values in a table, and DELETE removes records from a table. Data retrieval using SQL is based on the SELECT statement. The SELECT statement is described in a later section on queries. SQL is a powerful language, but it does take some effort to learn it. In many cases it is appropriate to hide this complexity from users. This can be done using query-by-form and other techniques where users are asked for the necessary information, and then the query is generated for them automatically.
Relational Data Management Theory
Stations WellID 1 2 3 Samples WellID 1 1 2 2 2 3 3 Flags Flag u v
Well B-1 B-2 B-3
Elev 725 706 714
SampleID 1 2 3 4 5 6 7
X 1050 342 785
SampDate 2/3/96 5/8/96 11/4/95 2/3/96 5/8/96 2/3/96 5/8/96
Name not det detected
Parameters Param Parameter 1 As 2 Cl 3 pH
Y 681 880 1101
Order 1 2 3
Sampler JLG DWR JAM JLG DWR JLG CRS
Analyses SampleID Param 1 1 1 3 2 1 2 2 2 3 3 1 3 2 3 3 4 1 4 2 4 3 5 1 5 2 5 3 6 1 6 3 7 1 7 2 7 3
Value .05 6.8 .05 .05 6.7 3.7 9.1 5.2 2.1 8.4 5.3 1.4 7.2 5.8 .05 8.1 .05 .05 7.9
27
Flag u v u u v v v v v v v v v v u v u u v
Figure 8 - Environmental data in fifth normal form with simple keys and coded values
Data retrieval is based on the SQL SELECT statement. The basic SELECT statement syntax (the way it is written) is: SELECT field list FROM table list WHERE filter expression An example of this would be an SQL query to extract the names and depths of the borings (stations) deeper than 10 feet: SELECT StationName, Depth FROM Stations WHERE Depth > 10 The SELECT and FROM clauses are required. The WHERE clause is optional. Data retrieval in Access and other relational database managers such as Oracle and SQL Server is based on this language. Most queries are more complicated than this. The following is an example of a complicated multi-table query created by Enviro Data, a product from Geotech Computer Systems, Inc. This query outputs data to be used for a report of analytical results.
28
Relational Management and Display of Site Environmental Data
Figure 9 - Data display resulting from a query
SELECT DISTINCTROW Stations.StationName, Stations.StationType, Stations.Location_CX, Stations.Location_CY, Stations.GroundElevation, Stations.StationDate_D, Samples.SampleType, Samples.SampleTop, Samples.SampleBottom, Samples.Sampler, Samples.SampleDate_D, Parameters.ParameterNumber, Parameters.LongName, Analytic.Value, Analytic.AnalyticMethod, Analytic.AnalyticLevel, Analytic.ReportingUnits, Analytic.Flag, Analytic.Detect, Analytic.Problems, Analytic.AnalDate_D, Analytic.Lab, Analytic.ReAnalysis AS Expr1 FROM StationTypes INNER JOIN (Stations INNER JOIN (Samples INNER JOIN ([Parameters] INNER JOIN Analytic ON Parameters.ParameterNumber = Analytic.ParameterNumber) ON Samples.SampleNumber = Analytic.SampleNumber) ON Stations.StationNumber = Samples.StationNumber) ON StationTypes.StationType = Stations.StationType WHERE (((Stations.StationName) Between [Forms]![AnalyticReport]![StartingStation] And [Forms]![AnalyticReport]![EndingStation]) AND ((Samples.SampleTop) Between [Forms]![AnalyticReport]![LowerElev] And [Forms]![AnalyticReport]![UpperElev]) AND ((Samples.SampleBottom) Between [Forms]![AnalyticReport]![LowerElev] And [Forms]![AnalyticReport]![UpperElev]) AND ((Parameters.ParameterNumber) Between [Forms]![AnalyticReport]![LowerParam] And [Forms]![AnalyticReport]![UpperParam])) ORDER BY Stations.StationName, Samples.SampleTop; Users are generally not interested in all of this complicated SQL code. They just want to see some data. Figure 9 shows the result of a query in Access.
Relational Data Management Theory
29
Figure 10 - Query that generated Figure 9
The SQL statement to generate Figure 9 is shown in Figure 10. It’s beyond most users to type queries like this. Access helps these users to create a query with a grid-type display. The user can drag and drop fields from the tables at the top to create their result. Figure 11 shows what the grid display looks like when the user is creating a query.
Figure 11 - Grid display of the query in Figure 10
30
Relational Management and Display of Site Environmental Data
Figure 12 - Query-by-form in the Enviro Data customized database system (This software screen and others not otherwise noted are courtesy of Geotech Computer Systems)
Even this is too much for some people. The complexity can be hidden even more in a customized system. The example in Figure 12 allows the users to enter selection criteria without having to worry too much about tables, fields, and so on. Then if users click on List they will get a query display similar to the one in Figure 9. In this system, they can also take the results of the query (the code behind this form creates an SQL statement just like the one above) and do other things such as create maps and reports.
BENEFITS OF NORMALIZATION The data normalization process can take a lot of effort. Why bother? The answer is that normalizing your data can provide several benefits, including greater data integrity, increased storage efficiency, and more flexible data retrieval.
Data integrity A normalized data model improves data integrity in several ways. The most important is referential integrity. This is an aspect of managing the data that is provided by the data management software. With referential integrity, relationships can be defined which require that a record in one table must be present before a record in a related table can be added. Usually the required record is a parent record (the “one” side of the one-to-many), which is necessary before a child record (the “many” side) can be entered. An example would be preventing you from entering a sample from a well before you have entered the well. More importantly, referential integrity can prevent deletion of a record when there are related records that depend on it, and this can be a great tool to prevent “orphan” records in the database. A good introduction to referential integrity in Access can be found in Harkins (2001b).
Relational Data Management Theory
31
A related concept is entity integrity. This requires that every row must be uniquely identified, meaning that each primary key value in the table must be unique and non-null. It is always a good idea for tables to have a primary key field, and that entity integrity be enforced. Another contributor to data integrity is the use of lookup tables, which is a capability of the relational database design. Lookup tables are tables where values that are repeated in the database are replaced by codes in the main data tables, and another table contains the codes and the lookups. An example is types of stations in the Stations table. Values for this field might include “soil boring,” “monitoring well,” and so on. These could be replaced by codes like “s” and “m,” respectively, in the main table, and the lookup table would contain the translation between the code and the value. It is much more likely that a user will mistype “detected” than “2.” Errors are even less likely if they are picking the lookup value off a list. They can still pick the wrong one, but at least errors from misspelling will be minimized.
Efficient storage Despite the increase in organization, the data after normalization as shown in Figure 8 actually takes less room than the original flat file. The data normalization process helps remove redundant data from the database. Because of this it may take less space to store the data. The original flat file contained 293 characters (not including field names) while the Fifth Normal Form version has only 254 characters, a decrease of 13% with no decrease in information content. The decrease in size can be quite a bit more dramatic with larger data sets. For example, in the new normalized model adding data for another analysis would take up only about 6 characters, where in the old model it would be about 34 characters, a savings in size of 82%.
Flexible retrieval When the data is stored in a word processor or spreadsheet, the output is usually a direct representation of the way the data is stored in the file. In a relational database manager, the output is usually based on a query, which provides an opportunity to organize and format the data a different way for each retrieval need.
AUTOMATED NORMALIZATION Normalization of a database is such an important process that some database programs have tools to assist with the normalization process. For example, Microsoft Access has a Table Analyzer Wizard that can help you split a table into multiple normalized tables based on the data content. To illustrate this, Figure 13 shows a very simple flat file of groundwater data with several stations, sample dates, and parameters. This file is already in First Normal Form, with no repeating groups, which is the starting point for the Wizard. The Table Analyzer Wizard was then run on this table. Figure 14 shows the result of this analysis. The Wizard has created three tables, one with the value, one with the station and sample date, and one with the parameter, units, and flag. It has also added several key fields to join the tables. The user can then modify the software’s choices using a drag and drop interface. The Wizard then steps you through a process for correcting typographical errors to improve the normalization process. For comparison, Figure 15 shows how someone familiar with the data would normalize the tables, creating primary tables for stations, samples, and analyses, and a lookup table for parameters. (Actually, units and flags should probably be lookup tables too.)
32
Relational Management and Display of Site Environmental Data
Figure 13 - Data set for the Table Analyzer Wizard
Figure 14 - Wizard analysis of groundwater data
Relational Data Management Theory
33
Figure 15 - Manual normalization of groundwater data
For this data set, the Wizard didn’t do very well, missing the basic hierarchical nature of the data. With other data sets with different variability in the data it might do better. In general the best plan is for people familiar with the data and the project needs to design the fields, tables, and relationships based on a thorough understanding of the natural relationships between the elements being modeled in the database. But the fact that Microsoft has provided a tool to assist with the normalization process highlights the importance of this process in database design.
CHAPTER 4 DATA CONTENT
A database can contain just about anything. The previous chapter discussed what a database is and how the data can be stored. A bigger issue is what data the database will contain. It is also important to make sure that database users understand the data and use it properly.
DATA CONTENT OVERVIEW The data contained in an EDMS can come from a wide variety of sources and can be used in many different ways. For some published examples, see Sara (1974), especially chapters 7 and 11. There is a wide range of data needs for the different projects performed by the typical environmental organization. With projects ranging from large sites with hundreds of wells and borings to small service stations with minimal data, and staffing levels from dozens of people on one project to many projects for one person, the variability of the data content needed for environmental projects is very great. Even once a data management system is in place, additional data items may be identified over time, as other groups and individuals are identified and become involved in the data management system, and as different project needs arise. Since different groups have differing data storage needs, a primary design goal of the data management system must be to accommodate this diversity while fostering standardization where possible. Sets of needs must be considered during design and development to avoid unnecessary limitations for any group. While there is always a trade-off between flexibility, ease of use, and development cost in creating a data management system, a well-designed data model for information storage can be worth the initial expense. Typical data components of an EDMS include four areas. The first three are project related: Project Technical Data, Project Administrative Data, and Project Document Data. The fourth, Reference Data, is not specific to an individual project, but in many cases has relationships to project-specific data. Often project data is the same as site data, so the terms are used interchangeably here. Figure 16 shows the main data components for the EDMS. This is just one way of organizing data. There are probably other equally valid ways of assigning a hierarchy to the data. Different data components are covered here in different levels of detail. The primary focus of this book is on project technical data, and especially data related to site investigation and remediation. This data is described briefly here, and in greater detail throughout the book. Other data components are covered briefly in this section, and have been well discussed in other books and articles. For each category of data, the major data components will be listed, along with, in many cases, comments about data management aspects of those components.
36
Relational Management and Display of Site Environmental Data
Enterprise
Project
Technical
Administrative
Reference
Documents
Figure 16 - Overview of the EDMS data model
PROJECT TECHNICAL DATA This category covers the technical information used by project managers, engineers, and geoscientists in managing, investigating, and remediating sites of environmental concern. Many acronyms are used in this section. The environmental business loves acronyms. You can find the meanings of these acronyms in Appendix A.
Product information and hazardous materials In an operating facility, tracking of materials within and moving through the facility can be very important from a health and safety perspective. This ranges from purchasing of materials through storage, use, and disposal. There are a number of aspects of this process where data management technology can be of value. The challenge is that in many cases the data management process must be integrated with the process management within the facility. Material information – The information to be stored about the materials themselves is roughly the same for all materials in the facility. Examples include general product information; materials management data; FIFRA labeling; customer usage; chemical information; and MSDS management, including creation, access, and updating/maintenance. Material usage – Another set of data is related to the usage of the materials, including shelf life tracking; recycling and waste minimization and reporting; and allegation and incident reports related to materials. Also included would be information related to keeping the materials under control, including source inventory information; LDR; pollution prevention; TSCA and asbestos management; and exception reports.
Hazardous wastes When hazardous materials move from something useful in the facility to something to be disposed of, they become hazardous wastes. This ranges from things like used batteries to toxic mixed wastes (affectionately referred to with names like “trimethyl death”). Waste handling – Safe handling and disposal of hazardous chemicals and other substances involves careful tracking of inventories and the shipping process. There is a significant recordkeeping component to working with this material. Data items include waste facility permit applications; waste accumulation and storage information (usually for discrete items like batteries); and waste stream information (more for materials that are part of a continuous process).
Data Content
37
If you put a drop of wine in a gallon of hazardous waste, you get a gallon of hazardous waste. If you put a drop of hazardous waste in a gallon of wine, you get a gallon of hazardous waste. Rich (1996) Waste disposal – If the waste is to be disposed onsite, then detailed records must be kept of the process, and data management can be an important component. If the waste is to be shipped offsite, then waste manifesting; waste shipping; NFPA/HMIS waste labels; and hazardous waste reports are important issues.
Environmental releases Inevitably, undesirable materials make it from the facility into the environment, and the EDMS can help track and report these releases. Types of releases – Water quality and drinking water management cover issues related to supposedly clean water. Wastewater management and pretreatment management cover water that is to be released beyond the facility. Once the material makes it into the groundwater, then groundwater management becomes important as discussed in the next section. Release issues – Regarding the releases themselves, issues include emissions management; air emissions inventory; NPDES permits; discharges and stormwater monitoring; stormwater runoff management; leak detection and LDAR; toxic chemical releases and TRI; and exceedence monitoring and reporting.
Site investigation and remediation The data gathered for site investigation and remediation is the major topic of this book. This data will be discussed briefly here, and in much more detail in other places in the book, especially Chapter 11 and Appendix B. Site data – Site data covers general information about the project such as location; type (organics, metals, radiologic, mixed, etc.); ownership; status; and so on. Other data to be stored at the site level includes QA and sampling plans; surveys; field data status; various other components of project status information; and much other information below the site level. Geology and hydrogeology – Geology and hydrogeology data includes surface geology information such as geologic maps, as well as subsurface information from wells, borings, and perhaps geophysical surveys. There are two kinds of geology/hydrogeology data for a site. The first could be considered micro data. This data is related to a particular boring or surface sample, and includes physical, lithological, and stratigraphic information. This information can be considered part of the sampling process, and can be included in the chemistry portion of the EDMS. The other type of data can be considered macro. This information covers more than an individual sample. It would include unit distribution and thickness maps; outcrop geology; facies maps (where appropriate); hydraulic head maps; and other data that is site-wide or nearly so. Stations – This is information related to locations of sampling such as monitoring wells; borings; surface water and air monitoring locations; and other sampling locations. In the EDMS, station data is separated into sample-related data and other data. The sample-related data is discussed below. Other station-related data includes surveys; boring logs; wellbore construction; and stratigraphy and lithology on a boring basis (as opposed to by sample). Some primary data elements for stations can cause confusion, including station coordinates and depths. For example, the XY coordinates used to describe station locations are based on some projection from latitude-longitude, or measurements on the surface of the earth from some (hopefully) known point (see Chapter 22). Either way, the meaning of the coordinates depends on reference information about the projection system of known points. Another example is depths,
38
Relational Management and Display of Site Environmental Data
When control equipment fails, it will do so during inspection. Rich (1996) which are sometimes stored as measured depths, and other times as elevations above sea level. With both of these examples, interpretation of numbers in fields in the database depends on knowledge of some other information. Samples – Information gathered for each sample includes sample date; frequency; depth; matrix; and many other things. This information is gathered from primary documents generated as part of the sampling process, such as the sample report and the Chain of Custody (COC). Quality control (QC) samples are an integral part of the sampling process for most environmental projects. An area where data management systems have great potential is in the interface between the field sampling event, the laboratory, and the centralized database; and for event planning; field management; and data checking. Analyses – The samples are then analyzed in the field and/or in the laboratory for chemical, physical, geotechnical, and sometimes geophysical parameters. QC data is important at the analyses level as well.
Cartography This category includes map-related data such as site maps; coordinates; air and satellite photos; and topography. It is a broad category. For example, site maps can include detailed maps of the site as well as regional maps showing the relationship of the site to geographic or other features. Implementation of this category in a way that is accessible to all the EDMS users requires providing some map display and perhaps editing capabilities as part of the system. This can be done by integrating an external mapping program such as ArcView, MapInfo, or Enviro Spase, or by inserting a map control (software object) like GeoObjects or MapObjects into the Access database. Data displayed on maps presented this way could include not only the base map information such as the site outline; buildings; etc. but also data from the EDMS, including well locations; sample counts; analytical values; and physical measurements like hydraulic head.
Coverages Some site data is discrete, that is, it is gathered at and represents a specific location in space. Other data represents a continuous variable across all or part of the site, and this type of data is referred to as a coverage. Examples of coverage data include surface elevation or the result of geophysical surveys such as gravity or resistivity. Often a single variable is sampled at discrete locations selected to represent a surface or volume. These locations can be on a regular grid or at irregular locations to facilitate sampling. In the first case, the locations can be stored implicitly by specifying the grid geometry and then listing the values. In the second case the coordinates need to be stored with each observation.
Models Models are spatial data that result from calculations. The EDMS can work with models in two ways. The first is to store and output data to be used in the modeling process. An example would be to select data and export an XYZ file of the current concentration of some constituent for use in a contouring program. This feature should be a standard part of the EDMS. The other model component of the EDMS is storage and display of the results of modeling. Most modeling projects result in a graphical output such as a contour map or three-dimensional display, or a numerical result. Usually these results can be saved as images in a file in vector (lines,
Data Content
39
polygons, etc.) or raster (pixels) format, or in numeric fields in tables. These graphics files or other results can be imported into the EDMS and displayed upon request. Examples of models often used in environmental projects include air dispersion; surface water and watershed configuration and flow; conceptual hydrologic; groundwater fate and transport; subsurface (2-D and 3-D) contouring; 3-D modeling and visualization; statistics, geostatistics, and sampling plan analysis; cross sections; volume estimates; and resource thicknesses and stripping ratios. Some of these are covered in more detail in later chapters.
Other issues Other data elements – The list of possible data elements to store for remediation projects is almost endless. Other data elements and capabilities include toxicity studies and toxicology; risk assessment; risk analysis and risk management; biology and ecology; SCADA; remediation planning and remedial plans; design calculations; and geotechnical designs. Summary information – Often it is useful to be able to see how many items of a particular kind of data are stored in the database, such as how many wells there are at a particular site, or how many arsenic analyses in the last year exceeded regulatory limits. There are two ways to provide this information, live or canned. In live count generation, the records are counted in real time as the result of a query. This is the most flexible approach, since the counts can be the result of nearly any query, and usually the most accurate, since the results are generated each time from the base data. With canned count generation, the counting is done separately from the display, and the results stored in a table. This is useful when the counts will be used often, and doing the counting takes a long time. It has the disadvantage that the counts can become “stale” when the primary data changes and the counts are not updated. Some systems use a combination of both approaches.
PROJECT ADMINISTRATIVE DATA Project administrative data covers a wide variety of types of data related to managing and administering projects. Examples of project administrative and management data elements include general site information; project management data; health and safety; and employee information. Some of this data consists of numbers and text items that should be stored using database management software. Other parts of this data are more document oriented, and might be a better fit in a document management system as discussed in a later section.
Site information Project overview – This could include site profile (overview) reports and descriptions; status reports and other general project information; and process descriptions and history. Ownership and permits – Items in this category include ownership and related administrative information; property transfers and related contact information; and information about permitting, permit management and tracking. Infrastructure – Possible infrastructure data elements include building and equipment information such as type; location; and operating history. Operations and maintenance – This is a broad area with much opportunity for data management assistance. This includes data related to requirements management; certifications, inspections, and audits; QA planning and documentation; continuous improvement and performance management; lockout/tagout surveys for equipment (where equipment that is not working properly is removed from service); energy analysis; emergency response; hazard analysis and tracking; collection systems; and residuals and biosolids management
40
Relational Management and Display of Site Environmental Data
Laboratory operations – If the site has one or more onsite laboratories, this would include data related to the LIMS; lab data checking; EDDs; and Web data delivery.
Project management Project management information can be stored as part of the general site database, or in specialized project management or accounting software. Budgets – Typical information in this category includes planned expenditures vs. actual expenditures; progress (% complete); costs on an actual and a unit basis, including EH&S (employee health and safety) expenses; sampling and analysis costs; and fee reports. Schedules – Schedule items can be regulatory items such as report due dates and other deadlines; or engineering items like work breakdown and critical path items. Reimbursement and fund recovery – For some projects it is possible to obtain partial reimbursement for remediation expenses from various funds such as state trust funds. The database can help track this process. Emission reduction credits – Facilities are required to meet certain emission criteria, and to encourage them to meet or even exceed these criteria they can be issued emission reduction credits that can be sold to facilities that are having trouble meeting their criteria. The second facility can then use these credits to minimize penalties. Customers and vendors – This would include information such as customer’s and vendor’s names and contact information; purchasing data; and electronic procurement of products and services. Other issues – There are many other project management data items that could be stored in the database. These include project review reports, records of status meetings, and so on.
Health and safety Tracking employee health and training is another fruitful area for a computerized data management approach. Because the health and safety of people is at stake, great care must be taken in data gathering, manipulation, and reporting of health and safety information. Facility information – Information to be tracked at the facility level includes process safety; hazard assessments; workplace safety plans and data; fire control and alarm systems; safety inspections and audits; and OSHA reports. Employee information – There is a large amount of information in this category that needs to be tracked, including safety, confined space, and other specialized training; accident/illness/injury tracking and reporting; individual exposure monitoring; and workers’ compensation and disability coverage and claims. General health and information – This category includes industrial hygiene; occupational medicine; environmental risk factors; toxicology and epidemiology; and monitoring of the health exposure and status of onsite and offsite individuals.
Personnel/project team Project staff – Data categories for project staff include the health and safety information discussed above, as well as organization information; personnel information; recruiting, hiring, and human resource information; demographics; training management and records; and certifications and fit tests. Others – Often information similar to that stored for project staff must be stored for others such as contractor and perhaps laboratory personnel.
Data Content
41
Incident tracking and reporting Despite the best efforts of project personnel, incidents of various types occur. The database system can help minimize incidents, and then track them when they do occur. Planning – Data to be stored prior to an incident includes emergency plans and information on emergency resources. Responding – This item covers things such as emergency management information and mutual aid (cooperation on emergency response). Tracking – This includes incident tracking; investigation and notification; “near miss” tracking; spill tracking; agency reports; and complaint tracking and response.
Regulatory status Regulatory status information for a facility or project includes quality assurance program plans; corrective action plans and corrective actions; rulings; limits (target values) and reporting times; Phase 1, Form R, and SARA (Superfund) reporting; Clean Air Act management; right to know management; project oversight and procedures; ROD and other limits; and certifications (such as ISO 9000 and 14000).
Multi-plant management Large organizations often have a large number of facilities with various environmental issues, each with a different level of urgency. Even big companies have limited resources, both financial and personnel-wise, and are unable to pay full attention to all of the needs of all of the facilities. Unfortunately, this can lead to a “brush-fire” approach to project management, where the projects generating the most heat get the most attention. Methods exist to evaluate facilities and prioritize those needing the most attention, with the intention of dealing with issues before they become serious problems.
Public affairs, lobbying, legislative activities The response of the public to site activities is increasing greatly in importance, with activities such as environmental justice lawsuits requiring more and more attention from site and corporate staff. A proactive approach of providing unbiased information to the public can, in some cases, minimize public outcry. A data management system can help organize these activities. Likewise, contacts with legislators and regulators can be of value, and can be managed using the database system.
PROJECT DOCUMENT DATA In terms of total volume of data, the project document data is probably the largest amount of data that will need to be managed within any organization. It may also be the most diverse from the viewpoint of its current physical configuration and its location. Options for storing document data are discussed in a later section of this chapter. The distinction between this category and the previous one about administrative data is that the previous data elements are part of the operations process, while items in this category are records of what happened. These items include compliance plans and reports; investigative reports (results); agreements; actual vs. planned activities; correspondence; drawings; and document management, control, and change tracking.
42
Relational Management and Display of Site Environmental Data
REFERENCE DATA Often it is desirable to provide access to technical reference information, both project-specific and general, on a timely basis. This includes boilerplate letters and standardized report templates. Reference data looks similar to project document data in its physical appearance, but is not associated with a specific project. Storing this type of information in a centralized location can be a great time saver in getting the work out. The storage options for this data are mostly the same as those for project document data. One difference is that some of the components of this data type may be links to external data sources, rather than data stored within the organization. An example of this might be a reference source for updated government regulations accessed from the Internet via the company intranet. Other than that, the reference data will usually be handled in the same way as project document data.
Administrative General administrative data can be similar to project data, but for the enterprise rather than for a specific project. This data includes timesheets; expense reports; project and task numbers; policies; correspondence numbers; and test reports. This category can also include information about shared resources, both internal and external, such as personnel and equipment availability; equipment service records; contractors; consultants; and rental equipment.
Regulatory Regulatory compliance information can be either general or project-specific. The project specific information was discussed above. In many cases it is helpful to store general regulatory compliance data in the database, including regulatory limits; reporting time guidelines; analyte suites; and regulatory databases (federal, state, local), including copies of regulations and decisions, like the Code of Federal Regulations. Regulatory alerts and regulatory issue tracking can also be helped with the use of a database.
Documentation Documentation at the enterprise level includes the information needed to run projects and the organization itself, from both the technical and administrative perspective. Technical information – Reference information in this category includes design information and formulas; materials engineering information; engineering guidelines; and other reference data. Enterprise financial information – Just as it is important to track schedules and budgets for each project, it is important to track this information for the enterprise. This is especially true for finances, with databases important for tracking accounting information, requests for proposals and/or bids; purchase orders and AFEs; employee expenses; and so on. Document resources – Sometimes a significant amount of time can be saved by maintaining a library of template documents that can be modified for use on specific projects. Examples include boilerplate letters and standardized reports. QA data – This category includes manuals; standard operating procedures for office, facility, and field use; and other quality documents. Quality management information can be general or specific. General information can include Standard Operating Procedures and other enterprise quality documents. Specific information can be project wide, such as a Quality Assurance Project Plan, or more specific, to the level of an individual analysis. Specific quality control information is covered in more detail in Chapter 15.
Data Content
43
News and information – It is important for staff members to have access to industry information such as industry news and events. Sometimes this is maintained in a database, but more commonly nowadays it is provided through links to external resources, especially Web pages managed by industry publications and organization.
DOCUMENT MANAGEMENT In all of the above categories, much of the data may be contained in existing documents, and often not in a structured database system. The biggest concern with this data is to provide a homogeneous view of very diverse data. The current physical format of this data ranges from digital files in a variety of formats (various generations of word processing, spreadsheet, and other files) through reports and oversized drawings. There are two primary issues that must be addressed in the system design: storage format and organization.
Storage options There are many different options for storing document data. There may be a part of this data that should be stored using database management software, but it usually consists largely of text documents, and to a lesser degree, diagrams and drawings. The text documents can be handled in five ways, depending on their current physical format and their importance. These options are presented in decreasing order of digital accessibility, meaning that the first options can be easily accessed, searched, edited, and so on, while the capability to do this becomes more limited with the later options. Many document management systems use a combination of these approaches for different types of documents. The first storage option, which applies to documents currently available in electronic form, is to keep them in digital form, either in their original format or in a portable document format such as Acrobat .pdf files (a portable document format from Adobe). Digital storage provides the greatest degree of flexibility in working with the documents, and provides relatively compact data storage (a few thousand bytes per page). The remaining four storage options apply to documents currently available only in hard copy form. The second and third options involve scanning the documents into digital form. In the second option, each document is scanned and then submitted to optical character recognition (OCR) to convert it to editable text. Current OCR software has an error rate of 1% or less, which sounds pretty good until you apply this to the 2000 or so characters on the average text page which gives 20 errors per page. Running the document through a spell-checker can catch many of these errors, but in order to have a usable document, it really should be manually edited as well. This can be very time-consuming if there are a large number of documents to be managed. Also, some people are good at this kind of work and some aren’t, so the management of a project to perform this conversion can be difficult. This option is most appropriate for documents that are available only in hard copy, but must be stored in editable form for updates, changes, and so on. It also provides compact storage (a few thousand bytes per page) if the scanned images are discarded after the conversion process. The third storage option, which is the second involving scanning, omits all or most of the OCR step. Instead of converting the documents to text and saving the text, the scanned images of the documents are saved and made available to users. In order to be able to search the documents, the images are usually associated with an index of keywords in the document. These keywords can be provided manually by reviewers, or automatically using OCR and specialized indexing software. Then when users want to find a particular document they can search the keyword index, locate the document they want, and then view it on the screen or print it. This option is most appropriate when the documents must be available online, but will be used for reference and not edited. It requires a large amount of storage. The theoretical storage size for an 8½ by 11 inch black and
44
Relational Management and Display of Site Environmental Data
white page at 300 dots per inch is about one megabyte, but with compression, the actual size is usually a few hundred thousand bytes, which is about a hundred times larger than the corresponding text document. The fourth storage option is to retain the documents in hard copy, and provide a keyword index similar to that in the third option. Creating the keyword index requires manually reviewing the documents and entering the keywords into a database. This may be less time-consuming than scanning and indexing. When the user wants a document, he or she can scan the keyword index to identify it, and then go and physically find it to work with it. This option requires very little digital storage, but requires a large amount of physical storage for the hard copies, and is the least efficient for users to retrieve documents. It is the best choice for documents that must be available, but will be accessed rarely. There is a final option for documents that do not justify any of the four previous options. That is to throw them away. On a practical level this is the most attractive for many documents, but for legal and other reasons is difficult to implement. A formal document retention policy with time limits on document retention can be a great help in determining which documents to destroy and when. For oversized documents, the options for storage are similar to those for page-sized documents, but the tools for managing them are more limited. For example, for a document in Word format, anyone in the organization can open it in Word, view it, and print it. For a drawing in AutoCAD format, the user must have AutoCAD, or at least a specialized CAD viewing program, in order to access it. To further complicate the issue, that drawing is probably a figure that should be associated with a text document. Part of the decision process for selecting document storage tools should include an evaluation of the ability of those tools to handle both regular and oversized documents, if both types are used in the organization.
Organization and display Once the decision has been made about the storage format(s) for the documents, then the organization of the documents must be addressed. Organization of the documents covers how they are grouped, where they are stored, and how they are presented. One possibility for grouping the documents is to organize them by type of document. Another is to divide them into quality-related and quality-unrelated documents. Documents that are in a digital format should be stored on a server in a location visible to all users with a legitimate need to see them. They can be stored in individual files or in a document management system, depending on the presentation system selected. The presentation of the data depends mostly on the software chosen. The options available for the type of data considered here include a structured database management system, a specialized document management system, the operating system file system, or a hypertext system such as the hypertext markup language (HTML) used for the Internet and the company intranet. A database management system can be used for document storage if an add-in is used to help manage the images of scanned pages. This approach is best when very flexible searching of the keywords and other data is the primary consideration. Specialized document management systems are programs designed to overcome many of the limitations of digital document storage. They usually have data capture interfaces designed for efficient scanning of large numbers of hard-copy documents. They may store the documents in a proprietary data structure or in a standard format such as .tif (tagged image file format). Some can handle oversized documents such as drawings and maps. They provide indexing capabilities to assist users in finding documents, although they vary in the way the indices work and in the amount of OCR that is done as part of the indexing. Some provide manual keyword entry only, while others will OCR some or all of the text and create or at least start the index for you. Document management systems work best when there is a large number of source documents with a similar
Data Content
45
physical configuration, such as multi-page printed reports. They do less well when the source documents are very diverse (printed reports, hard copy and digital drawings and maps, word processor and spreadsheet digital documents, etc.) as in many environmental organizations. This software might be useful if the decision is made to scan and index a large volume of printed documents, in which case it might be used to assist with the data capture. The file system built into the operating system provides a way of physically storing documents of all kinds. With modern operating systems, it provides at least some multi-user access to the data via network file sharing. This is the system currently used in most organizations for storing and retrieving digital files, and many may wish to continue with this approach for file storage. (For example, an intranet approach relies heavily on the operating system file storage capabilities.) Where this system breaks down is in retrieving documents from a large group, since the retrieval depends on traversing the directory and file structure and locating documents by a (hopefully meaningful) file name. In this case, a better system is needed for organizing and relating documents. Hypertext systems have been growing in popularity as a powerful, flexible way of presenting large amounts of data where there may be great variation in the types of relationships between data elements. Some examples of hypertext systems include the Hypercard program for the Apple Macintosh and the Help System for Microsoft Windows. A more recent and widely known example of hypertext systems is the HTML system used by the World Wide Web (and its close relative, the intranet). All hypertext systems have as a common element the hyperlink, which is a pointer from one document into another, usually based on highlighted words or images. The principal advantage of this approach is that it facilitates easy movement of the user from one resource to another. The main disadvantage is the effort required to convert documents to HTML, and the time that it takes to set up an intuitive system of links. This approach is rapidly becoming the primary document storage system within many companies.
PART TWO - SYSTEM DESIGN AND IMPLEMENTATION
CHAPTER 5 GENERAL DESIGN ISSUES
The success of a data management task usually depends on the tool used for that task. The theoretical physicist Stephen Hawking is quoted as saying, “When all you have is a hammer, everything looks like a nail.” This is as true in data management as in anything else. People who like to use a word processor or a spreadsheet program are likely to use the tool they are familiar with to manage their data. But just as a hammer is not the right tool to tighten a screw, a spreadsheet is not the right tool to manage a large and complicated database. A database management program should be used instead. This section discusses the design of the database management tool, and how the design can influence the success of the project.
DATABASE MANAGEMENT SOFTWARE Database management programs fall into two categories, desktop and client-server. The use of the two different types and decisions about where the data will be located are discussed in the next section. This section will discuss database applications themselves and briefly discuss the features and benefits of the programs. The major database software vendors put a large amount of effort into expanding and improving their products, so these descriptions are a snapshot in time. For an overview of desktop and Web-based database software, see Ross et al. (2001). Older database systems were, for the most part, based on dBase, or at least the dBase file format. dBase started in the early days of DOS, and was originally released as dBase II because that sounded more mature than calling it 1.0. If anyone tells you that they have been doing databases since dBase 1 you know they are bluffing. dBase was an interpreted application, meaning that the code was translated into machine language (compiled) each time it was run, which was slow on those early computers. This created a market for dBase compilers, of which FoxPro was the most popular. Both used a similar data format in which each data table was called a database file or .dbf. Relationships were defined in code, rather than as part of the data model. Much data has been, and still is in some cases, managed in this format. These files were designed for single-user desktop use, although locking capabilities were added in later versions of the software to allow shared use. Nowadays Microsoft Access dominates the desktop database market. This program provides a good combination of ease of use for beginners and power for experts. It is widely available, either as a stand-alone product or as part of the Office desktop suite. Additional information on Access can be found in books by Dunn (1994), Jennings (1995), and others, and especially in journals such as PC Magazine and Access/Visual Basic Advisor. Access has a feature that is common to almost all successful database programs, which is a programming language that allows users to
50
Relational Management and Display of Site Environmental Data
automate tasks, or even build complete programs using the database software. In the case of Access, there are actually two programming models, a macro language that is falling out of favor, and a programming language. The programming language is called Visual Basic for Applications (VBA), and is a fairly complete development environment. Since Access is a desktop database, it has limitations relative to larger systems. Experience has shown that for practical use, the software starts to have performance problems when the largest table in a database starts to reach a half million to a million records. Access allows multiple users to share a database, and no programming is required to implement this, but a dozen or so concurrent users is an upper limit for heavy usage database scenarios. Access is available either as a stand-alone product or as part of the Microsoft Office Suite. An alternative to Access is Paradox from Corel (www.corel.com). This is a programmable, relational database system, and is available as part of the Corel Office Suite. Paradox is a capable tool suitable for a complex database project, but the greater acceptance of Access makes Paradox an unlikely choice in the environmental business where file sharing is common, and Access is widespread. The next step up from Access for many organizations is Microsoft SQL Server. This is a fullscale client-server system with robust security and a larger capacity than Access. It is moderately priced and relatively easy to use (for enterprise software), and increases the capacity to several million records. It is easy to attach an Access front end (user interface) to a SQL Server back end (data storage), so the transition to SQL Server is relatively easy when the data outgrows Access. This connection can be done using ODBC (Open DataBase Connectivity) or other connection methods. For even larger installations, Oracle or IBM’s DB2 offer industrial-strength data storage, but with a price and learning curve to match. These can also be connected to the desktop using connection methods like ODBC, and one front-end application can be set up to talk to data in these databases, as well as to data in Access. Using this approach it is possible to create one user interface that can work with data in all of the different database systems. A new category of database software that is beginning to appear is open-source software. Open-source software refers to programs where the source code for the software is freely available, and the software itself is often free as well. This type of software is popular for Internet applications, and includes such popular programs as the Linux operating system and Apache Web server. Two open-source database programs are PostgreSQL and MySQL (Jepson, 2001). These programs are not yet as robust as commercial database systems, but are improving rapidly. They are available in commercial, supported versions as well as the free open-source versions, so they are starting to become options for enterprise database storage. And you can’t beat the price. Another new category of database software is Web-based programs. These programs run in a browser rather than on the desktop, and are paid for with a monthly fee. Current versions of these programs are limited to a flat-file design, which makes them unsuitable for the complex, relational nature of most environmental data, but they might have application in some aspects of Web data delivery. Examples of this type of software include QuickBase from the authors of the popular Quicken and QuickBooks financial software (www.quickbase.com), and Caspio Bridge (www.caspio.com).
DATABASE LOCATION OPTIONS A key decision in designing a data management system is where the data will reside. Related to this are a variety of issues, including what hardware and software will provide the necessary functionality, who will be responsible for data entry and editing, and who will be responsible for backup of the database.
General Design Issues
51
Stand-alone The simplest design for a database location is stand-alone. In this design, the data and the software to manage it reside on the computer of one user. That computer may or may not be on a network, but all of the functionality of the database system is local to that machine. The hardware and software requirements of a system like this are modest, requiring only one computer and one license for the database management software. The software does not need to provide access for more than one user at a time. One person is in control of the computer, software, and data. For small projects, especially one-person projects, this type of design is often adequate. For larger projects where many people need access to the data, the single individual keeping the data can become a bottleneck. This is particularly true when the retrievals required are large or complicated. The person responsible for the data can end up spending most or all of his or her time responding to data requests from users. When the data management requirements grow beyond that point, the stand-alone system no longer meets the needs of the project team, and a better design is required.
Shared file Generally the next step beyond a stand-alone system is a shared file system. In a shared file system, the server (or any computer) stores the database on its hard drive like any other file. Clients access the file using database software on their computers the same way they would open any other file on the server. The operating system on the server makes the file available. The database software on the client computer is responsible for handling access to the database by multiple users. An example of this design would be a system in which multiple users have Microsoft Access on their computers, and the database file, which has an extension of .mdb, resides on a server, which could be running Windows 95/98/ME or NT/2000/XP. When one or more users is working in the database file, their copy of Access maintains a second file on the server called a lock file. This file, which has an extension of .ldb, keeps track of which users are using the database and what objects in the database may be locked at any particular time. This design works well for a modest number of users in the database at once, providing adequate performance for a dozen or so users at any given time, and for databases up to a few hundred thousand records.
Client-server When the load on the database increases to the point where a shared file system no longer provides adequate performance, the next step is a client-server system. In this design, a data manager program runs on the server, providing access to the data through a system process. One computer is designated the server, and it holds the data management software and the data itself. This system may also be used as the workstation of an individual user, but in high-volume situations this is not recommended. More commonly, the server computer is placed on a network with the computers of the users, which are referred to as clients. The software on the server works with software on the client computers to provide access to the data. The following diagram covers the internal workings of a client-server EDMS. It contains two parts, the Access component at the top and the SQL Server part at the bottom. In discussing the EDMS, the Access component is sometimes called the user interface, since that is the part of the system that users see, but in fact both Access and SQL Server have user interfaces. The Access user interface has been customized to make it easy for the EDMS users to manage the data in ways useful to them. The SQL Server user interface is used without modifications (as provided by Microsoft) for data administration tasks. Between these user interfaces are a number of pieces that work together to provide the data management capabilities of the system.
52
Relational Management and Display of Site Environmental Data
Access User Interface Lookup Table Maintenance Electronic Import
Manual Entry
Table View
Data Review
Record Counts
Maps
Subset Creation
Graphs
Formatted Reports Fmt1 Fmt2
Client Server
Selection Scr. Access Queries / Modules Access Attachments
Selection Screen Access Queries / Modules Access Attachments
Security System Read/Write
Security System Read Only Server Tables
Volume Maint.
File Export Fmt1 Fmt2
Subset Database Backup / Restore
Server Volume
SQL Server / Oracle User Interface Figure 17 - Client-server EDMS data flow diagram
Discussion of this diagram will start at the bottom and work toward the top, since this order starts with the least complicated parts (from the user’s perspective, not the complexity of the software code) and moves to the more complicated parts. That means starting on the SQL Server side and working toward the client side. This sequence provides the most orderly view of the system. In this diagram, the part with a gray background represents SQL Server, and the rest of the box is Access. The basic foundation of the server side is the SQL Server volume, which is actually a file on the server hard drive that contains the data. The size of this volume is set when the database is set up, and can be changed by the administrator as necessary. Unlike many computer files, it will not grow automatically as data is added. Someone needs to monitor the volume size and the amount of data and keep them in synch. The software works this way because the location and structure of the file on the hard drive is carefully managed by the SQL Server software to provide maximum performance (speed of query execution). The database tables are within the SQL Server volume. These tables are similar in function and appearance to the tables in Access. They contain all of the data in the system, usually in normalized database form. The data in the tables is manipulated through SQL queries passed to the SQL Server software via the ODBC link from the clients. Also stored in the SQL Server volume can be triggers and stored procedures to increase performance in the client-server system and to enforce referential integrity. If they wish, users can see the tables in the SQL Server volume through the database window in Access, but their ability to manipulate the data depends on the privileges that they have through the security system. A System Administrator should back up data in the SQL Server tables on a regular basis (at least daily). The interface between the EDMS and the SQL Server tables is through a security system that is partly implemented in SQL Server and partly in Access. Most users should have read-only permission for working with the data; that is, they will be able to view data but not change it. A small group of users, called data administrators, should be able to modify data, which will include importing and entering data, changing data, and deleting data.
General Design Issues
53
The actual connection between Access and SQL Server is done through attachments. Attachments in Access allow Access to see data that is located somewhere other than the current Access .mdb file as if it were in the current database. This is the usual way of connecting to SQL Server, and also allows us to provide the flexibility of attaching to other data sources. Once the attachments are in place, the client interaction with the database is through Access queries, either alone or in combination with modules, which are programs written in VBA. Various queries provide access to different subsets of the data for different purposes. Modules are used when it is necessary to work with the data procedurally, that is, line by line, instead of the whole query as a set.
Distributed The database designs described above were geared toward an organization where the data management is centralized, that is, where the data management activities are performed in one central location, usually on a local area network (LAN). With environmental data management, this is not always the case. Often the data for a particular facility must be available both at the facility and at the central office. The situation becomes even more complicated when the central office must manage and share data with multiple remote sites. This requires that some or all of the data be made available at multiple locations. The following sections describe three ways to do this: wide-area networks, distributed databases with replication, and remote communication with subsets. The factors that determine which solution is best for an organization include the amount of data to be managed, how fresh the data needs to be at the remote locations, and whether full-time communication between the facilities is available and the speed of that communication. Wide-area networks – In situations where a full-time, high-speed communication link is or can be made available (at a cost which is reasonable for the project), a wide-area network (WAN) is often the best choice. From the point of view of the people using it, the WAN hardware and software connect the computers just as with a LAN. The difference is that instead of all of the computers being connected directly through a local Ethernet or Token Ring network, some are connected through long-distance data lines of some sort. Often there are LANs at the different locations connected together over the WAN. The connection between the LANs is usually done with routers on either end of the longdistance line. The router looks at the data traffic passing over the network, and data packets which have a destination on the other LAN are routed across the WAN to that LAN, where they continue on to their destination. There are several options for the long-distance lines forming the WAN between the LANs. This discussion will cover some popular existing and emerging technologies. This is a rapidly changing industry, and new technologies are appearing regularly. At the high end of connectivity options are dedicated analog solutions such as T1 (or in cases of very high data volume, T3) or frame relay. These services are connected full-time, and provide high to moderate speeds ranging from 56 kilobits per second (kbps) to 1 megabit per second (mbps) or more. These services can cost $1000 per month or more. This is proven technology, and is available, at a cost, to nearly any facility. Recently, newer digital services have become available for some areas. Integrated Services Digital Network (ISDN) provides 128 kbps for around $100 per month. Digital Subscriber Line (DSL) provides connectivity ranging in speed from 256 kbps to 1.5 mbps or more. Prices can be as low as $40 per month, so this service can be a real bargain, but service is limited to a fairly short distance from the telephone company central office, so it’s not available to many locations. Cable modems promise to give DSL a run for its money. It’s not widely available right now, especially for business locations, and when it is it will have more of a focus on residential rather than business since that is where cable is currently connected.
54
Relational Management and Display of Site Environmental Data
Another option is standard telephone lines and analog modems. This is sometimes called POTS (plain old telephone system). This provides 56 kbps, or more with modem pooling, and the connection is made on demand. The cost is relatively low ($15 to 30 per month) and is available everywhere. In order to have WAN-level connectivity, you should have a full-time connection of about 1 mbps or faster. If the connection speed available is less than this, another approach should be used. Distributed databases with replication – There are several situations where a client-server connection over a WAN is not the best solution. One is where the connection speed is too low for real-time access. The second is where the data volume is extremely high, and local copies make more sense. In this situation, distributed databases can make sense. In this design, copies of the database are placed on local servers at each facility, and users work with the local copies of the data. This is an efficient use of computer resources, but raises the very important issue of currency of the data. When data is entered or changed in one copy, the other copy is no longer the most current, and, at some point, the changes must be transferred between the copies. Most high-end database programs, and now some low end ones, can do this automatically at specified intervals. This is called replication. Generally, the database manager software is smart enough to move only the changed data (sometimes called “dirty” records) rather than the whole database. Problems can occur when users make simultaneous changes to the same records at different locations, so this approach rapidly becomes complicated. Remote communication with subsets – Often it is valuable for users to be able to work with part of the database remotely. This is particularly useful when the communication line is slow. In this scenario, users call in with their remote computers and attach to the main database, and either work with it that way or download subsets for local use. In some software this is as easy as selecting data using the standard selection screen, then instructing the EDMS to create a subset. This subset can be created on the user’s computer, then the user can hang up, attach to the local subset, then use the EDMS in the usual way, working with the subset. This works for retrieving data, but not as well for data entry or editing, unless a way is provided for data entered into the subset to be uploaded to the main database.
Internet/intranet Tools are now available to provide access to data stored in relational database managers using a Web browser interface. At the time of this writing, in the opinion of the author, these tools are not yet ready for use as the primary interface in a sophisticated EDMS package. Specifically, the technology to provide an intuitive user interface with real-time feedback is too costly or impossible to build with current Web development tools. Vendors are working on implementing that capability, but the technology is not currently ready for prime time, at least for everyday users, and the current technology of choice is client-server. It is now feasible, however, to provide a Web interface with limited functionality for specific applications. For example, it is not difficult to provide a public page with summaries of environmental data for plants. The more “canned” the retrieval is, the easier it is to implement in a browser interface, although allowing some user selection is not difficult. In the near future, tools like Dynamic HTML, Active Server Pages, and better Java applets, combined with universal high-speed connections, will make it much easier to provide an interactive user interface, hosted in a Web browser. At that time, EDMS vendors will certainly provide this type of interface to their client’s databases. The following figure shows a view of three different spectra provided by the Internet and related technologies. There are probably other ways of looking at it, but this view provides a framework for discussing products and services and their presentation to users. In these days of multi-tiered applications, this diagram is somewhat of an over-simplification, but it serves the purpose here.
General Design Issues
Local
55
Global Applications
StandAlone
Shared Files
Client- WebServer Enabled
WebBased
Data Proprietary
Commercial
Public Domain
Users Desktops
Laptops
PDAs, etc.
Public Portals
Figure 18 - The Internet spectrum
The overall range of the diagram in Figure 18 is from Local on the left to Global on the right. This range is divided into three spectra for this discussion. The three spectra, which are separate but not unrelated, are applications, data, and users. Applications – Desktop computer usage started with stand-alone applications. A program and the data that it used were installed on one computer, which was not attached to others. With the advent of local area networks (LANs), and in some organizations wide-area networks (WANs), it became possible to share files, with the application running on the local desktop and with the data residing on a file server. As software evolved and data volumes grew, software was installed on both the local machine (client) and the server, with the user interface operating locally, and data storage, retrieval, and sometimes business logic operating on the server. With the advent of the Internet and the World Wide Web, sharing on a much broader scale is possible. The application can reside either on the client computer and communicate with the Web, or it can run on a Web server. The first type of application can be called Web-enabled. An example of this is an email program that resides locally, but talks through the Web. Another example would be a virusscanning program that operates locally but goes to the Web to update its virus signature files. The second type of application can be called Web-based. An example of this would be a browser-based stock trading application. Many commercial applications still operate in the range between stand-alone and client server. There is now a movement of commercial software to the right in this spectrum, to Web-enabled or Web-based applications, probably starting with Web-enabling current programs, and then perhaps evolving to a thin-client, browser-based model over time. This migration can be done with various parts of the applications at different rates depending on the costs and benefits at each stage. New technologies like Microsoft’s .NET initiative are helping accelerate this trend. Data – Most environmental database users currently work mostly with data that they generate themselves. Their base map data is usually based on CAD drawings that they create, and the rest of their project data comes from field measurements and laboratory analyses, which the data manager (or their client) pays for and owns. This puts them to the left of the spectrum in the above figure. Many vendors now offer both base map and other data, either on the Web or on CD-ROM, which might be of value to many users. Likewise, government agencies are making more and more data, spatial and non-spatial, available, often for free. As vendors evolve their software and Web presence, they can work toward integrating this data into their offerings. For example, software could be used to load a USGS or Census Bureau base map, and then display sites of environmental concern obtained from the EPA. Several software companies provide tools to make it possible to
56
Relational Management and Display of Site Environmental Data
serve up this type of data from a modified Web server. Revenue can be obtained from purchase or rental of the application, as well as from access to the data. Users – The World Wide Web has opened up a whole new world of options for computing platforms. These range from the traditional desktop computers and laptops through personal digital assistants (PDAs), which may be connected via wireless modem, to Web portals and other public access devices. Desktops and laptops can run present and future software, and, as most are becoming connected to the Internet, will be able to support any of the computing models discussed above. PDAs and other portable devices promise to provide a high level of portability and connectivity, which may require re-thinking data delivery and display. Already there are companies that integrate global positioning systems (GPS) with PDAs and map data to show you where you are. Other possible applications include field data gathering and delivery, and a number of organizations provide this capability. Web portals include public Internet access (such as in libraries and coffee shops) as well as other Internet-enabled devices like public phones. This brings up the possibility that applications (and data) may run on a device not owned by or controlled by the client, and argues for a thin-client approach. This is all food for thought as we try to envision the evolution of environmental software products and services (see Chapter 27). What is clear is that the options for delivery of applications and data have broadened significantly, and must be considered in planning for future needs.
Multi-tiered The evolution of the Internet and distributed computing has led to a new deployment model called “multi-tiered.” The three most common tiers are the presentation level, the business logic level, and the data storage level. Each level might run on a different computer. For example, the presentation level displayed to the user might run on a client computer, using either client-server software or a Web browser. The business logic level might enforce the data integrity and other rules of the database, and could reside on a server or Web server computer. Finally, the data itself could reside on a database server computer. Separating the tiers can provide benefits for both the design and operation of the system.
DISTRIBUTED VS. CENTRALIZED DATABASES An important decision in implementing a data management system for an organization performing environmental projects for multiple sites is whether the databases should be distributed or centralized. This is particularly true when the requirements for various uses of the data are taken into consideration. This issue will be discussed here from two perspectives. The first perspective to be discussed will be that of the data, and the second will be that of the organization. From the perspective of the data and the applications, the options of distributed vs. centralized databases are illustrated in Figures 19 and 20. Clearly it is easier for an application to be connected to a centralized, open database than a diverse assortment of data sources. The downside is the effort required to set up and maintain a centralized data repository.
General Design Issues
Mapping
?
Project Management
?
?
GIS Coverages
Statistics
Reporting
Validation
?
Lab Deliverables
Spreadsheets
CAD Files
Legacy Systems
?
ASCII Files
Word Proc. Files
Chain of Custody
Field Notebooks
Hard Copy Files
Regulatory Reports
?
?
?
Graphing
Planning
Web Access
Figure 19 - Connection to diverse, distributed data sources
Mapping
Validation
Reporting
Statistics
Centralized Open Database
Graphing
Project Management
Web Access
Planning
Figure 20 - Connection to a centralized open database
57
58
Relational Management and Display of Site Environmental Data
CLIENT
The Dilemma
The Solution
Site 1
Consultant Consultant Consultant Consultant Consultant 3A 3B 2B 1 2A
Site 4
Consultant 1 + Geotech
!"#$%& Other Access Spreadsheet Do It '()( EDMS Other Yourself EDMS
Consultant 4
Labs Web etc. !"#$%&*'()(
Figure 21 - Distributed vs. centralized databases
The choice of distributed vs. centralized databases can also be viewed from the perspective of the organization. This is illustrated in Figure 21. The left side of the diagram shows the way the data for environmental projects has traditionally been managed. The client, such as an industrial company, owns several sites with environmental issues. One or more consultants, labeled C1, C2, etc., manage each site, and each consultant may manage the project from various offices, such as C2A, C2B, etc. Each consultant office might use a different tool to manage the data. For example, for Site 1, consultant C1 may use an Excel spreadsheet. Consultant C2, working on a different part of the project, or on the same issues at a different time, may use a home-built database. Other consultants working on different sites use a wide variety of different tools. If people in the client organization, either in the head office or at one of the sites, want some data from one monitoring event, it is very difficult for them to know where to look. Contrast this with the right side of the diagram. In this scenario, all of the client’s data is managed in a centralized, open database. The data may be managed by the client, or by a consultant, but the data can be accessed by anyone given permission to do so. There are huge savings in efficiency, because everyone knows where the data is and how to get at it. The difficult challenge is getting the data into the centralized database before the benefits can be realized.
General Design Issues
59
Figure 22 - Example of a simplified logical data model
THE DATA MODEL The data model for a data management system is the structure of the tables and fields that contain the data. Creating a robust data model is one of the most important steps in building a successful data management system (Walls, 1999). If you are building a data management system from scratch, you need to get this part right first, as best you can, before you proceed with the user interface design and system construction. Many software designers work with data models at two levels. The logical data model (Figure 22) describes, at a conceptual level, the data content for the system. The lines between the boxes represent the relationships in the model. The physical data model (Figure 23) describes in detail exactly how the data will be stored, with names, data types, and sizes for all of the fields in each table, along with the relationships (key fields which join the tables) between the tables. The overall scope of the logical data model should be identified as early in the design process as possible. This is particularly true when the project is to be implemented in stages. This allows identification of the interactions between the different parts of the system so that dependencies can be planned for as part of the detailed design for each subset of the data as that subset is implemented. Then the physical data model for the subset can be designed along with the user interface for that subset. The following sections describe the data structure and content for a relational EDMS. This structure and content is based on a commercial system developed and marketed by Geotech Computer Systems, Inc. called Enviro Data. Because this is a working system that has managed hundreds of databases and millions of records of site environmental investigation and monitoring data, it seems like a good starting point for discussing the issues related to a data model for storing this type of data.
60
Relational Management and Display of Site Environmental Data
Figure 23 - Table and field display from a physical data model
Data structure The structure of a relational EDMS, or of any database for that matter, should, as closely as possible, reflect the physical realities of the data being stored. For environmental site data, samples are taken at specific locations, at certain times, depths, and/or heights, and then analyzed for certain physical and chemical parameters. This section describes the tables and relationships used to model this usage pattern. The section after this describes in some detail the data elements and exactly how they are used so that the data accurately reflects what happened. Tables – The data model for storing site environmental data consists of three types of tables: primary tables, lookup tables, and utility tables. The primary tables contain the data of interest. The lookup tables contain codes and their expanded values that are used in the primary tables to save space and encourage consistency. Sometimes the lookups contain other useful information for the data elements that are represented by the coded values. The utility tables provide a place to store various data items, often related to the operation and maintenance of the system. Often these tables are not related directly to the primary tables. For the most part, the primary data being stored in the EDMS has a series of one-to-many (also known as parent-child or hierarchical) relationships. It is particularly fortunate that these relationships are one-to-many rather than many-to-many, since one-to-many relationships are handled well by the relational data model, and many-to-many are not. (Many-to-many relationships can be handled in the relational data model. They require adding another table to track the links between the two tables. This table is sometimes called a join table. We don’t have to worry about that here.)
General Design Issues
61
The primary tables in this system are called Sites, Stations, Samples, and Analyses. The detailed content of these tables is described below. Sites contains information about each facility being managed in the system. Stations stores data for each location where samples are taken, such as monitoring wells and soil borings. (Note that what is called a station in this discussion is called a site in some system designs.) Samples represents each physical sample or monitoring event at specific stations, and Analyses contains specific observed values or analytical results from the samples. Relationships – The hierarchical relationships between the tables are obvious. Each site can have one or more stations, each station has one or more samples, and each sample is analyzed for one or more, often many, constituents. But each sulfate measurement corresponds to one specific sampling event for one specific location for one specific site. The lookup relationships are one-to-many also, with the “one” side being the lookup table and the “many” side being the primary table. For example, there is one entry in the StationTypes table for monitoring wells, with a code of “mw,” but there can be (and usually are) many monitoring wells in the Stations table.
Data content This section will discuss briefly the data content of the example EDMS. This material will be covered in greater detail in Appendix B. Sites – A Site is a facility or project that will be treated as a unit. Some projects may be treated as more than one site, and sometimes a site can be more than one facility, but the use of the site terminology should be consistent within the database, or at least for each project. Some people refer to a sampling location as a site, but in this discussion we will call that a station. Stations – A Station is a location of observation. Examples of stations include soil borings, monitoring wells, surface water monitoring stations, soil and stream sediment sample locations, air monitoring stations, and weather stations. A station can be a location that is persistent, such as a monitoring well which is sampled regularly, or can be the location of a single sampling event. For stations that are sampled at different elevations (such as a soil boring), the location of the station is the surface location for the boring, and the elevation or depth component is part of the sampling event. Samples – A Sample is a unique sampling event or observation for a station. Each station can be sampled at various depths (such as with a soil boring), at various dates and times (such as with a monitoring well), or, less commonly, both. Observations, which may or may not accompany a physical sample, can be taken at a station at a particular time, and in this model would be considered part of a sample event. Analyses – An Analysis is the observed value of a parameter related to a sample. This term is intended to be interpreted broadly, and not to be limited to chemical analyses. For example, field parameters such as pH, temperature, and turbidity also are considered analyses. This would also include operating parameters of environmental concern such as flow, volume, and so on. Lookups – A lookup table is a table that contains codes that are used in the main data tables, and the expanded values of those codes that are used for selection and display. Utilities – The system may contain tables for tracking internal information not directly related to the primary tables. These utility tables are important to the software developers and maybe the system and data administrators, but can usually be ignored by the users.
DATA ACCESS REQUIREMENTS The user interface provides a number of data manipulation functions, some of which are read/write and the rest are read-only.
62
Relational Management and Display of Site Environmental Data
Read-write The functions that require read/write access to the database are: Electronic import – This function allows data administrators to import analytical and other data. Initially the data formats supported will be the three formats defined in the Data Transfer Standard. Other import formats may be added as needed. This is shown in Figure 17 as a singleheaded arrow going into the database, but in reality there is also a small flow of data the other way as the module checks for valid data. Manual entry – The hope is that the majority of the data that will be put in the system will be in digital format that can be imported without retyping. However, there will probably be some data which will need to be manually entered and edited, and this function will allow data administrators to make those entries and changes. Editing – Data administrators will sometimes need to change data in the database. Such changes must be done with great care and be fully documented. Lookup table maintenance – One of the purposes of the lookup tables is to standardize the entries to a limited number of choices, but there will certainly be a need for those data tables to evolve over time. This feature allows the data administrators to edit those tables. A procedure will be developed for reviewing and approving those changes before entry. Verification and validation – Either as part of the import process or separately, data validators will need to add or change validation flags based on their work. Data review – Data review should accompany data import and entry, and can be done independently as well. This function allows data administrators to look at data and modify its data review flag as appropriate, such as after validation.
Read-only The functions that require read-only access to the database are: Record counts – This function is a useful guide in making selections. It should provide the number of selected items whenever a selection criterion is changed. Table view – This generalized display capability allows users to view the data that they have selected. This might be all of the output they need, or they might wish to proceed to another output option, once they have confirmed that they have selected correctly. They can also use this screen to copy the data to the clipboard or save it to a file for use in another application. Formatted reports – Reports suitable for printing can be generated from the selection screen. Different reports could be displayed depending on the data element selected. Maps – The results of the selection can be displayed on a map, perhaps with the value of a constituent for each station drawn next to that station and a colored dot representing the value. See Chapter 22 for more information on mapping. Graphs – The most basic implementation of this feature allows users to draw a graph of constituent values as a function of time for the selected data. They should be able to graph multiple constituents for one station or one constituent for several stations. More advanced graphing is also possible as described in Chapter 20. Subset creation – Users should be able to select a subset of the main database and export it to an Access database. This might be useful for providing the data to others, or to work with the subset when a network connection to the database is unavailable or slow. File export – This function allows users to export data in a format suitable for use in other software needing data from the EDMS. Formats need to be provided for the data needs of the other software. Direct connection without export-import is also possible.
General Design Issues
63
GOVERNMENT EDMS SYSTEMS A number of government agencies have developed systems for managing site environmental data. This section describes some of the systems that are most widely used. STORET (www.epa.gov/storet) – STORET (short for STOrage and RETrieval) is EPA’s repository for water quality, biological, and physical data. It is used by EPA and other federal agencies, state environmental agencies, universities, private citizens, and others. It is one of EPA’s two data management systems containing water quality information for the nation's waters. The other system, the Legacy Data Center, or LDC, contains historical water quality data dating back to the early part of the 20th century and collected up to the end of 1998. It is being phased out in favor of STORET. STORET contains data collected beginning in 1999, along with older data that has been properly documented and migrated from the LDC. Both LDC and STORET contain raw biological, chemical, and physical data for surface and groundwater collected by federal, state, and local agencies, Indian tribes, volunteer groups, academics, and others. All 50 states, territories, and jurisdictions of the U.S., along with portions of Canada and Mexico, are represented in these systems. Each sampling result is accompanied by information on where the sample was taken, when the sample was gathered, the medium sampled, the name of the organization that sponsored the monitoring, why the data was gathered, and much other information. The LDC and STORET are Web-enabled, so users can browse both systems interactively or create files to be downloaded to their computer for further use. CERCLIS (www.epa.gov/superfund/sites/cursites) – CERCLIS is a database that contains the official inventory of Superfund hazardous waste sites. It contains information on hazardous waste sites, site inspections, preliminary assessments, and remediation of hazardous waste sites. The EPA provides online access to CERCLIS data. Additionally, standard CERCLIS site reports can be downloaded to a personal computer. CERCLIS is a database and not an EDMS, but can be of value in EDMS projects. IRIS (www.epa.gov/iriswebp/iris/index.html) – The Integrated Risk Information System, prepared and maintained by the EPA, is an electronic database containing information on human health effects that may result from exposure to various chemicals in the environment. The IRIS system is primarily a collection of computer files covering individual chemicals. These chemical files contain descriptive and quantitative information on oral reference doses and inhalation reference concentrations for chronic non-carcinogenic health effects, and hazard identification, oral slope factors, and oral and inhalation unit risks for carcinogenic effects. It is a database and not an EDMS, but can be of value in EDMS projects. ERPIMS (www.afcee.brooks.af.mil/ms/msc_irp.htm) – The Environmental Resources Program Information Management System (ERPIMS, formerly IRPIMS) is the U.S. Air Force system for validation and management of data from environmental projects at all Air Force bases. The project is managed by the Air Force Center for Environmental Excellence (AFCEE) at Brooks Air Force Base in Texas. ERPIMS contains analytical chemistry samples, tests, and results as well as hydrogeological information, site/location descriptions, and monitoring well characteristics. AFCEE maintains ERPTools/PC, a Windows-based software package that has been developed to help Air Force contractors in collection and entry of their data, validation, and quality control. Many ERPIMS data fields are filled by codes that have been assigned by AFCEE. These codes are compiled into lists, and each list is the set of legal values for a certain field in the database. Air Force contractors use ERPTools/PC to prepare the data, including comparing data to these lists, and then submit it to the main ERPIMS database at Brooks. IRDMIS (aec.army.mil/prod/usaec/rmd/im/imass.htm) – The Installation Restoration Data Management Information System (IRDMIS) supports the technical and managerial requirements of the Army's Installation Restoration Program (IRP) and other environmental efforts of the U.S. Army Environmental Center (USAEC, formerly the U.S. Toxic and Hazardous Materials Agency). (Don’t confuse this AEC with the Atomic Energy Commission, which is now the Department of
64
Relational Management and Display of Site Environmental Data
Energy.) Since 1975, more than 15 million data records have been collected and stored in IRDMIS with information collected from over 100 Army installations. IRDMIS users can enter, validate, store, and retrieve the Army’s geographic; geological and hydrological; sampling; chemical; and physical analysis information. The system covers all aspects of the data life cycle, including complete data entry and validation software using USAEC and CLP QA/QC methods; a Web site for data submission and distribution; and an Oracle RDMS with menu-driven user interface for standardized reports, geographical plots, and plume modeling. It provides a fully integrated information network of data status and disposition for USAEC project officers, chemists, geologists, contracted laboratories, and other parties, and supports Geographical Information Systems and other third-party software. USGS Water Resources (http://water.usgs.gov/nwis) – This is a set of Web pages that provide access to water resources data collected at about 1.5 million locations in all 50 states, the District of Columbia, and Puerto Rico. The U.S. Geological Survey investigates the occurrence, quantity, quality, distribution, and movement of surface and groundwater, and provides the data to the public. Online access to data on this site includes real-time data for selected surface water, groundwater, and water quality sites; descriptive site information for all sites with links to all available water data for individual sites; water flow and levels in streams, lakes, and springs; water levels in wells; and chemical and physical data for streams, lakes, springs, and wells. Site visitors can easily select data and retrieve it for on-screen display or save it to a file for further processing.
OTHER ISSUES Creating and maintaining an environmental database is a serious undertaking. In addition to the activities directly related to maintaining the data itself, there are a number of issues related to the database system that must be considered.
Scalability Databases grow with time. You should make sure that the tool you select for managing your environmental data can grow with your needs. If you store your data in a spreadsheet program, when the number of lines of data exceeds the capacity of the spreadsheet, you will need to start another file, and then you can’t easily work with all of your data. If you store your data in a standalone database manager program like Access, when your data grows you can relatively easily migrate to a more powerful database manager like SQL Server or Oracle. The ability of software and hardware to handle tasks of different sizes is called scalability, and this requirement should be part of your planning if there is any chance your project will grow over time.
Security The cost of building a large environmental database can be hundreds of thousands of dollars or more. Protect this investment from loss. Ensure that only authorized individuals can get access to the database. Make adequate backups frequently. Be sure that the people who are working with the database are adequately trained so that they do a good job of getting clean data into the database, and that the data stays there and stays clean. Instill an attitude of protecting the database and keeping its quality up so that people can feel comfortable using it.
Access and permissions Most database manager programs provide a system for limiting who can use a database, and what actions they can perform. Some have more than one way of doing this. Be sure to set up and
General Design Issues
65
use an access control system that fits the needs of your organization. This may not be easy. You will have to walk a thin line between protecting your data and letting people do what they need to do. Sometimes it’s better to start off more restrictive than you think you need to, and then grant more permissions over time, than to be lenient and then need to tighten up, since people react better to getting more power rather than less. Also be aware that security and access limitations are easier to implement and manage in a client-server system than in a stand-alone system, so if you want high security, choose SQL Server or Oracle over Access for the back-end.
Activity tracking To guarantee the quality of the data in the database, it is important to track what changes are made to the data, when they are made, who made them, and why they were made. A simple activity tracking system would include an ActivityLog table in the database to allow data administrators to track data modifications. On exit from any of the data modification screens, including importing, editing, or reviewing, an activity log screen will appear. The program reports the name of the data administrator and the activity date. The data administrator must enter a description of the activity, and the name of the site that was modified. The screen should not close until an entry has been made. Figure 24 shows an example of a screen for this type of simple system. The system should also provide a way to select and display the activity log. Figure 25 shows an example of a selection screen and report of activity data. In this example, the log can be filtered on Administrator name, Activity Date, or Site. If no filters are entered, the entire log is displayed. Another option is a more elaborate system that keeps copies of any data that is changed. This is sometimes called a shadow system or audit log. In this type of system, when someone changes a record in a table, a copy of the unchanged record is stored in a shadow table, and then the change is made in the main table. Since most EDMS activity usually does not involve a lot of changes, this does not increase the storage as much as it might appear, but it does significantly increase the complexity of the software.
Figure 24 - Simple screen for tracking database activity
66
Relational Management and Display of Site Environmental Data
Figure 25 - Output of activity log data
Database maintenance There are a number of activities that must be performed on an ongoing or at least occasional basis to keep an EDMS up and running. These include: Backup – Backing up data in the database is discussed in Chapter 15, but must be kept in mind as part of ongoing database maintenance. Upgrades – Both commercial and custom software should be upgraded on a regular basis. These upgrades may be required due to a change in the software platform (operating system, database software) or to add features and fix bugs. A system should be implemented so that all users of the EDMS receive the latest version of the software in a timely fashion. For large enterprises with a large number of users, automated tools are available to assist the system administrator with distributing upgrades to all of the appropriate computers without having to visit each one. Web-based tools are beginning to appear that provide the same functionality for all users of software programs that support this feature. Either of these approaches can be a great time saver for a large enterprise system. Other maintenance – Other maintenance activities are required, both on the client side and the server side. For example, on the client side, Access databases grow in size with use. You should occasionally compact your database files. You can do this on some set schedule, such as monthly, or when you notice that it has grown large, such as larger than 5 megabytes (5000 Kb). Occasionally problems will occur with Access databases due to power failures, system crashes, etc.
General Design Issues
67
When this happens, first exit Access, then shut down Windows, power down the computer, and restart. If you get errors in the database after that, you can have Access repair and compact the database. In the worst case (if repairing does not work), you should obtain a new copy of the database program from the original source, and restore your data file from a backup. System maintenance will be required on the server database as well, and will generally be performed by the system administrator with assistance from the vendor if necessary. These procedures include general maintenance of the server computer, user administration, database maintenance, and system backup. The database is expected to grow as new data is received for sites currently in the database, and as new sites are added. At some point in the future it will be necessary to expand the size of the device and the database to accommodate the increased volume of data which is anticipated. The system administrator should monitor the system to determine when the database size needs to be increased.
CHAPTER 6 DATABASE ELEMENTS
A number of elements make up an EDMS. These elements include the computer on the user’s desk, the software on that computer, the network hardware and software, and the database server computer. They also include the components of the database management system itself, such as files, tables, fields, and so on. This chapter covers the important elements from these two categories. This presentation focuses on how these objects are implemented in Access (for standalone use) and SQL Server (for client-server), two popular database products from Microsoft. A good overview of Access for both new and experienced database users can be found in Jennings (1995). More advanced users might be interested in Dunn (1994). More information on SQL Server can be found in Nath (1995); England and Stanley (1996); and England (1997). More information on database elements can be found in Dragan (2001), Gagnon (1998), Harkins (2001a, 2001b), Jepson (2001), and Ross et al. (2001).
HARDWARE AND SOFTWARE COMPONENTS A modern networked data management system consists of a number of hardware and software components. These items, which often come from different manufacturers and vendors, all must work together for the system to function properly.
The desktop computer It is obvious that in order to run a data management system, either client-server or stand-alone, you must have a computer, and the computer resources must be sufficient to run the software. Data management programs can be relatively large applications. In order to run a program like this you must have a computer capable of running the appropriate operating system such as Windows. This section describes the desktop hardware and software requirements for either a client-server or stand-alone database management system. Other than the network connection, the hardware requirements are the same.
DESKTOP HARDWARE The computer should have a large enough hard drive and enough random access memory (RAM) to be able to load the software and run it with adequate performance, and data management software can have relatively high requirements. For example, Microsoft Access has the greatest resource requirements of any of the Microsoft Office programs. At the time of this writing, the minimum and recommended computer specifications for adequate performance using the data management system are as shown in Figure 26.
70
Relational Management and Display of Site Environmental Data
Item Computer Hard drive Memory Removable storage Display Network Peripherals
Minimum 200 megahertz Pentium processor Adequate for software and local data storage, at least 1 gigabyte 64 megabytes RAM 3.5” floppy, CD-ROM
Recommended 500 to 1000 megahertz Pentium processor Adequate for software and local data storage, at least 1 gigabyte 128 megabytes RAM 3.5” floppy, CD-RW, Zip drive
VGA 800x600 10 megabits per second Printer
XGA 1024x768 or better 100 megabits per second High-speed printer, scanner
Figure 26 - Suggested hardware specifications
Probably the most important requirement is adequate random access memory (RAM), the chips that provide short-term storage of data. The amount of RAM should be increased on machines that are not providing acceptable performance. If increasing the RAM does not increase the performance to a level appropriate for that user’s specific needs, then replacing the computer with a faster one may be required. It is important to note that the hardware requirements to run the latest software, and the computer processing power of standard systems available at the store, both become greater over time. Computers that are more than three years or so old may be inadequate for running the latest version of the database software. A brand-new, powerful computer including a monitor and printer sells for $1000 or less, so it doesn’t make sense to limp along on an underpowered, flaky computer. Don’t be penny-wise and pound-foolish. Be sure that everyone has adequate computers for the work they do. It will save money in the long run. An important distinction to keep in mind is the difference between memory and storage. A computer has a certain amount of system memory or RAM. It also has a storage device such as a hard drive. Often people confuse the two, and say that their computer has 10 gigabytes of memory, when they mean disk storage.
DESKTOP SOFTWARE Several software components are required in order to run a relational database management system. These include the operating system, networking software (where appropriate), database management software, and the application.
Operating system Most systems used for data management run one of the Microsoft operating systems: Windows 95, 98, ME, or NT/2000/XP. All of these systems can run the same client data management software and perform pretty much the same. Apple Macintosh systems are present in some places, but are used mostly for graphic design and education, and have limited application for data management due to poor software availability. UNIX systems (including the popular open-source version, Linux) are becoming an increasingly viable possibility, with serious database systems like Oracle and DB2 now available for various types of UNIX.
Networking software If the data is to be managed with a shared-file or client-server system, or if the files containing a single-user database are to be stored on a file server computer, the client computer will need to run networking software to make the network interface card work. In some cases the networking software is part of the operating system. This is the case with a Windows network. In other cases the networking will be done with a separate software package. Examples include Novell Netware
Database Elements
71
and Banyan Vines. Either way, the networking software will generally be loaded during system startup, and after that can pretty much be ignored, except that network file server resources and network database server resources are available. This networking software is described in more detail in the next section.
Database management software The next software element in the database system is the database management software itself. Examples of this software are Microsoft Access, FoxPro, and Paradox. This software can be used by itself to manage the data, or with the help of a customized application as described in the next section. The database application provides the user interface (the menus and forms that the user sees) and can, in the case of a stand-alone or single-user system, also provide the data storage. In a client-server system, the database software on the client computer provides the user interface, and some or all of the data is stored on the database server computer somewhere else on the network. If the data to be managed is relatively simple, the database management software by itself is adequate for managing it. For example, a simple table of names and addresses can be created and data entered into it with a minimum of effort. As the data model becomes more complicated, and as the interaction between the database and external data sources becomes more involved, it can become increasingly difficult to perform the required activities using the tools of the software by itself. At that point a specialized application may be required.
Application When the complexity of the database or its interactions exceeds the capability of the generalpurpose database manager program, it is necessary to move to a specialized vertical market application. This refers to software specialized for a particular industry segment. An EDMS represents software of this type. This type of system is also referred to as COTS (commercial offthe-shelf) software. Usually the vertical market application will provide pre-configured tables and fields to store the data, import routines for data formats common in the industry, forms for editing the data, reports for printing selected data, and export formats for specific needs. Using off-theshelf EDMS software can give you a great head start in building and managing your database, relative to designing and building your own system.
The network Often the EDMS will run on a network so people can share the data. The network has hardware and software components, which are discussed in the following sections.
NETWORK HARDWARE The network on which the EDMS operates has three basic components, in addition to the computers themselves: network adapters, wiring, and hubs. These network hardware components are shown in Figure 27. The network adapters are printed circuit boards that are placed in slots in the client and server computers and provide the electronic connection between the computer and the network. The type of adapter card used depends on the kind of computer in which it is placed, and the type of network being used.
72
Relational Management and Display of Site Environmental Data
Clients
Network Adapter
Network Adapter
Network Adapter
Network Hub
Network Adapter
Server Figure 27 - The EDMS network hardware diagram
The wiring also depends on the type of network being used. The two most common types of wiring are twisted pair and coaxial, usually thin Ethernet. Twisted pair is becoming more common over time due to lower cost. Most twisted pair networks use Category 5 (sometimes called Cat5) cable, which is similar to standard telephone wiring, but of higher quality. There is usually a short cable that runs between the computer and a wall plate, wiring in the walls from the client’s or server’s office to a wiring closet, and then another cable from the wall plate or switch block in the wiring closet to the hub. The hub is a piece of hardware that takes the cables carrying data from the computers on the network and connects them together physically. Depending on the type of network and the number of computers, other hardware may be used in place of or in addition to the hub. This might include network switches or routers. The network can run at different speeds depending on the capability of the computers, network cards, hubs, wiring, and so on. Until recently 10 megabits per second was standard for local area networks (LANs), and 56 kilobits per second was common for wide-area networks (WANs). Increasingly, 100 megabits per second is being installed for LANs and 1 megabit per second or faster is used for WANs.
EDMS NETWORK SOFTWARE There are a number of software components required on both the client and server computers in order for the EDMS to operate. Included in this category is the operating system transport protocols and other software required just to make the computer and network work. The operating system and network software should be up and running before the EDMS is installed.
Database Elements
73
Clients
Access Front-end
Access Front-end
Access Front-end
ODBC Driver
ODBC Driver
ODBC Driver
SQL Queries Data In
Query Results Data Out
SQLServer Process SQLServer Data Storage
Backup and Restore
Server Figure 28 - The EDMS network software components
The major networked data management software components of the EDMS are discussed in this section from an external perspective, that is, looking at the various pieces and what they do, but not at the detailed internal workings of each. The important parts of the internal view, especially of the data management system, will be provided in later sections. On the client computers in a client-server system, the important components for data management provide the user interface and communication with the server. On the server, the software completes the communication and provides storage and manipulation of the data. For a stand-alone system, both parts run on the client computer. The diagram in Figure 28 shows the major data management software components for a client-server system, based on Access as a front-end and SQL Server as a back-end. On the client computers, the user interface for the EDMS can be provided by a database such as Microsoft Access, or can be written in a programming language like Visual Basic, Power Builder, Java, or C++. The advantage of using a database language is ease of development and flexibility. The advantage of a compiled language is code security, and perhaps speed, although speed is less of a distinguishing factor than it used to be. The main user interface components are forms and menus for soliciting user input and forms and reports for displaying output. Also provided by Access on the desktop are queries to manipulate data and macros and modules (both of which are types of programs) to control program
74
Relational Management and Display of Site Environmental Data
operation and perform various tasks. Customized components specific to the EDMS, if any, are contained in an Access .mdb file which is placed on the client computer during setup and which can be updated on a regular basis as modifications are made to the software. Through this interface, the user should be able to (with appropriate privileges) import and check data, select subsets of the data, and generate output, including tables, reports, graphs, and maps. To communicate data with the server, the Access software works with a driver, which is a specialized piece of software with specific capabilities. In a typical EDMS this driver uses a data transfer protocol called Open DataBase Connectivity (ODBC). The driver for communicating with SQL Server is provided by Microsoft as part of the Access software installation, although it may not be installed as part of the standard installation. Drivers for other server databases are available from various sources, often the vendor of the database software. There are two parts to the ODBC system in Windows. One part is ODBC administration, which can be accessed through the ODBC icon in Control Panel. This part provides central management of ODBC connections for all of the drivers that are installed. The second part consists of individual drivers for specific data sources. There are two kinds of ODBC drivers, single-tier and multi-tier. The single-tier drivers provide both the communication and data manipulation capabilities, and the data management software for that specific format itself is not required. Examples of single-tier drivers include the drivers for Access, dBase, and FoxPro data files. Multi-tier drivers provide the communication between the client and server, and work with the database management software on the server to provide data access. Examples of multi-tier drivers include the drivers for SQL Server and Oracle. The server side of the ODBC communication link is provided by software that runs on the server as an NT/2000/XP process. The SQL Server process listens for data requests from clients across the network via the ODBC link, executes queries locally on the server, and sends the results back to the requesting client. This step is very important, because the traffic across the network is minimized. The requests for data are in the form of SQL queries, which are a few hundred to a few thousand characters, and the data returned is whatever was asked for. In this way the user can query a small amount of data from a database with millions of records and the network traffic would be just a few thousand characters. Some EDMS software packages can work in either stand-alone or client-server mode. In the first case it uses a direct link to the Jet database engine when working with an Access database. In the second case, the EDMS uses the SQL Server multi-tier driver to communicate between the user interface in Access and SQL Server on the server. When users are attached to a local Access database, all of the processing and data flow occurs on the client computer. When connected to the server database the data comes from the server.
The server SERVER HARDWARE The third hardware component of the EDMS, besides client computers and the network, is the database server. This is a computer, usually a relatively powerful one, which contains the data and runs the server component of the data management software. Usually it runs an enterprise-grade operating system such as Windows NT/2000/XP or UNIX. In large organizations the server will be provided or operated by an Information Technology (IT) or similar group, while in smaller organizations data administrators or power users in the group will run it. The range of hardware used for servers, especially running Windows NT/2000/XP, is great. NT/2000/XP can run on a standard PC of the type purchased at discount or office supply stores. This is actually a good solution for small groups, especially when the application is not mission critical, meaning that if the database becomes unavailable for short periods of time the company won’t be shut down.
Database Elements
75
Figure 29 - Example administrative screen from Microsoft SQL Server
For an organization where the amount of use of the system is greater, or full-time availability is very important, a computer designed as a server, with redundant and hot-swappable (can be replaced without turning off the computer) components, is a better solution. This can increase the cost of the computer by a factor of two to ten or more, but may be justified depending on the cost of loss of availability.
SERVER SOFTWARE The client-based software components described above are those that users interact with. System administrators also interact with the server database user interface, which is software running on the server computer that allows maintenance of the database. These maintenance activities include regular backup of the data and occasional other maintenance activities including user and volume administration. Software is also available which allows many of these maintenance activities to be performed from computers remote from the server, if this is more convenient. An example screen from SQL Server is shown in Figure 29.
UNITS OF DATA STORAGE The smallest unit of information used by computers is the binary bit (short for BInary digiT). A bit is made up of one piece of data consisting of either a zero or a one, or more precisely, the electrical charge is on or off at that location in memory. All other types of data are composed of one or more bits. The next larger common unit of storage is the byte, which contains eight bits. One byte can represent one of 256 different possibilities (two raised to the eighth power). This allows a byte to represent any one of the characters of the alphabet, the numbers and punctuation symbols, or a large number of other characters. For example, the letter A (capital A) can be represented by the byte 01000001. How each character is coded depends on the coding convention used. The two most common are ASCII (American Standard Code for Information Interchange) used on personal
76
Relational Management and Display of Site Environmental Data
computers and workstations, and EBCDIC (Extended Binary Coded Decimal Interchange Code) used on some mainframes. The largest single piece of data that can be handled directly by a given processor is called a word. For an 8-bit machine, a word is the same as a byte. For a 16-bit system, a word is 16 bits long, and so on. A 32-bit processor is faster than a 16-bit processor of the same clock speed because it can process more data at once, since the word size is twice as big. For larger amounts of data, the amount of storage is generally referred to in terms of the number of bytes, usually in factors of a thousand (actually 1024, or 210). Thus one thousand bytes would be one kilobyte, one million would be one megabyte, one billion is one gigabyte, and one trillion is one terabyte. As memory, mass storage devices, and databases become larger, the last two terms are becoming increasingly important.
DATABASES AND FILES As discussed in Chapter 5, databases can be described by their logical data model, which focuses on data and relationships, and their physical data model, which is how the data is stored in the computer. All data in a modern computer is stored in files. Files are chunks of related data stored together on a disk drive such as a hard disk or floppy disk. The operating system takes care of managing the details of the files such as where they are located on the disk. Files have names, and files in DOS and Windows usually have a base name and an extension separated by a period, such as Mydata.dbf. The extension usually tells you what type of file it is. Older database systems often stored their data in the format of dBase, with an extension of .dbf. Access stores its data and programs in files with the extension of .mdb for Microsoft DataBase, and can store many tables and other objects in one file. Most Access developers build their applications with one .mdb file for the program information (queries, forms, reports, etc.) and another for the data (tables). Larger database applications have their data in an external database manager such as Oracle or SQL Server. The user does not see this data as files, but rather as a data source available across the network. If the front end is running in Access, they will still have the program .mdb either on their local hard drive or available on a network drive. If their user interface is a compiled program written in Visual Basic, C, or a similar language, it will have an extension of .exe. We will now look at the remaining parts of a database system from the point of view of a stand-alone Access database. The concepts are about the same for other database software packages. Access databases contain six primary objects. These are tables, queries, forms, reports, macros, and modules. These objects are described in the following sections.
TABLES (“DATABASES”) The basic element of storage in a relational database system is the table. Each table is a homogeneous set of rows of data describing one type of real-world object. In some older systems like dBase, each table was referred to as a database file. Current usage tends more toward considering the database as the set of related tables, rather than calling one table a database. Tables contain the following parts: Records – Each line in a table is called a record, row, entity, or tuple. For example, each boring or analysis would be a record in the appropriate table. Records are described in more detail below. Fields – Each data element within a record is called a field, column, or attribute. This represents a significant attribute of a real-world object, such as the elevation of a boring or the measured value of a constituent. Records are also described in more detail below.
Database Elements
77
Figure 30 - Join Properties form in Microsoft Access
Relationships – Data in different tables can be related to each other. For example, each analysis is related to a specific sample, which in turn is related to a specific boring. Relationships are usually based on key fields. The database manager can help in enforcing relationships using referential integrity, which requires that defined relationships be fulfilled according to the join type. Using this capability, it would be impossible to have an analysis for which there is no sample. Join types – A relationship between two tables is defined by a join. There are two kinds of joins, inner joins and outer joins. In an inner join, matching records must be present on both sides of the join. That means that if one of the tables has records that have no matching records in the other, they are not displayed. An outer join allows unmatched records to be displayed. It can be a left join or a right join, depending on which table will have unmatched records displayed. Figure 30 shows an example of defining an outer join in Access. In this example, a query has been created with the Sites and Stations tables. The join based on the SiteNumber field has been defined as an outer join, with all records from the Sites table being displayed, even if there are no corresponding records in the Stations table. This outer join is a left join. Figure 31 shows the result of this query. There are stations for Rad Industries and Forest Products Co., but none for Refining, Inc. Because of the outer join there is a record displayed for Refining, Inc. even though there are no stations.
Figure 31 - Result of an outer join query
78
Relational Management and Display of Site Environmental Data
FIELDS (COLUMNS) The fields within each record contain the data of each specific kind within that record. These are analogous to the way columns are often used in a spreadsheet, or the blanks to be filled out on a paper form. Data types – Each field has a data type, such as numeric (several possible types), character, date/time, yes/no, object, etc. The data type limits the content of the field to just that kind of data, although character fields can contain numbers and dates. You shouldn’t store numbers in a character field, though, if you want to treat them as numbers, such as performing arithmetic calculations on them. Character fields are the most common type of field. They may include letters, numbers, punctuation marks, and any other printable characters. Some typical character fields would be SiteName, SampleType, and so on. Numeric is for numbers on which calculations will be performed. They may be either positive or negative, and may include a decimal point. Numeric fields that might be found in an EDMS are GroundElevation, SampleTop, etc. Some systems break numbers down further into integer and floating point numbers of various degrees of precision. Generally this is only important if you are writing software, and less important if you are using commercial programs. It is important to note that Microsoft programs such as Excel and Access have an annoying feature (bug) that refuses to save trailing zeros, which are very important in tracking precision. If you open a new spreadsheet in Excel, type in 3.10, and press Enter, the zero will go away. You can change the formatting to get it back, but it’s not stored with the number. The best way around this is to store the number of decimals with each result value, and then format the number when it is displayed. Date is pretty obvious. Arithmetic calculations can often be performed on dates. For example, the fields SampleDate and AnalysisDate could be included in a table, and could be subtracted from each other to find the holding time. Date fields in older systems are often 8 characters long (MM/DD/YY), while more modern, year 2000 compliant systems are 10 characters (MM/DD/YYYY). There is some variability in the way that time is handled in data management systems. In some database files, such as dBase and FoxPro .dbf files, date and time are stored in separate fields. In others, such as Access .mdb files, both can be stored in one field, with the whole number representing the date and the decimal component containing the time. The dates in Access are stored as the number of days since 12/30/1899, and times as the fraction of the day starting at midnight, such that .5 is noon. The way dates are displayed traditionally varies from one part of the world to another, so as we go global, be careful. On Windows computers, the date display format is set in the operating system under Start/Settings/Control Panel/Regional Settings. Logical represents a yes/no (true/false) value. Logical fields are one byte long (although it actually takes only one bit to store a logical value). ConvertedValue could be a logical field that is true or false based on whether or not a value in the database has been converted from its original units. Data domain – Data within each field can be limited to a certain range. For example, pH could be limited to the range of 0 to 14. Comprehensive domain checking can be difficult to implement effectively, since in a normalized data model, pH is not stored in its own field, but in the same Value field that stores sulfate and benzene, which certainly can exceed 14. That means that this type of domain analysis usually requires programming. Value – Each field has a value, which can be some measured amount, some text attribute, etc. It is also possible that the value may be unknown or not exist, in which case the value can be set to Null. Be aware, however, that Null is not the same as zero, and is treated differently by the software.
Database Elements
79
Figure 32 - Oracle screen for setting field properties
Key fields – Within each table there should be one or more fields that make each record in the table unique. This might be some real-world attribute (such as laboratory sample number) or a synthetic key such as a counter assigned by the data management system. A primary key has a unique value for each record in the table. A field in one table that is a primary key in another table is called a foreign key, and need not be unique, such as on the “many” side of a one-to-many relationship. Simple keys, which are made up of one field, are usually preferable to compound keys made up of more than one field. Compound keys, and in fact any keys based on real data, are usually poor choices because they depend on the data, which may change. Figure 32 shows an Oracle screen for setting field properties.
RECORDS (ROWS) Once the tables and fields have been defined, the data is usually entered one record at a time. Each well in the Stations table or groundwater sample in the Samples table is a record. Often the size of a database is described by the number of records in its tables.
QUERIES (VIEWS) In Access, data manipulation is done using queries. Queries are based on SQL, and are given names and stored as objects, just like tables. The output of a query can be viewed directly in an editable, spreadsheet-like view, or can be used as the basis of a form or a report. Access has six types of queries: Select – This is the basic data retrieval query. Cross-tab – This is a specialized query for summarizing data.
80
Relational Management and Display of Site Environmental Data
Figure 33 - Simple data editing form
Make table – This query is used to retrieve data and place it into a new table. Update – This query changes data in an existing table. Append – This query type adds records to an existing table. Delete – These queries remove records from a table, and should be used with great care!
OTHER DATABASE OBJECTS The other types of database objects in an Access system are forms, reports, macros, and modules. Forms and reports are for entering and displaying data, while macros and modules are for automating operations.
Forms Forms in data management programs such as Access are generally used for entering, editing, or selecting data, although they can also be used as menus for selecting an activity. Forms for working with data use a table or a query as a data source.
Figure 34 - Advanced data editing form
Database Elements
81
Figure 35 - Example of a navigation form
Figure 33 shows an example of a simple form for editing data from one table. Data editing forms can be much more complicated than the previous example. Figure 34 shows a data editing form with many fields and a subform, which allows for many records in the related table to be displayed for each record in the main table. Figure 35 shows a form used for navigation. Users click on one of the gray rectangles with their mouse to open the appropriate form for what they want to do. Sometimes the data entry forms can be combined with navigation capabilities. The following form is mostly a data entry form, with data fields and a subform, but it also allows users to navigate to a specific record. They do this by opening a combo box, and selecting an item from the list. The form then takes them to that specific record. Forms are a very important part of a database system, since they are usually the main way users interact with the system.
Figure 36 - A form combining data entry and navigation
82
Relational Management and Display of Site Environmental Data
Figure 37 - Report of analytical data
Reports Reports are used for displaying data, usually for printing. Reports use a table or form as a data source, the same way that forms do. The main differences are that the data on reports cannot be edited, and reports can better handle large volumes of data using multiple pages. Figure 37 shows a typical report of analytical data. Reports will be covered in much more detail in Chapter 19.
Macros, modules, subroutines, and functions Much of the power in modern data management programs comes from the ability to program them for specific needs. Some programs, like Access, provide more than one way to tell the program what to do. The two ways in Access are macros and modules. Macros are like stored keystrokes, and are used to automate procedures. Modules are more like programs, and can also be used to automate activities. Modules have some advantages over macros. Many Access developers prefer modules to macros, but since macros are easier to learn, they are frequently used, especially by beginners. Microsoft encourages use of modules instead of macros for programming their applications, and suggests that support for macros may be removed in future versions. Figure 38 shows an Access macro in the macro-editing screen. This macro minimizes the current window and displays the STARTUP form.
Database Elements
83
Figure 38 - Access macro example
Modules provide the programming power behind Access. They are written in Access Basic, a variety of Visual Basic for Applications (VBA). VBA is a complete, powerful programming language that can do nearly anything that any programming language can do. VBA is customized to work with Access data, which makes it easy to write sophisticated applications to work with data in Access tables. Figure 39 shows the Access screen for editing a module. The subroutine shown re-spaces the print order in the Parameters table with an increment of five so new parameters can be inserted in between. A module can have two kinds of code in it, subroutines and functions (also known as “subs”), both of which are referred to as procedures. Both are written in VBA. The difference is that a function returns a value, and a sub does not. Otherwise they can do exactly the same thing.
Figure 39 - Access module example
84
Relational Management and Display of Site Environmental Data
Figure 40 - SQL Server screen for editing a trigger
Triggers and stored procedures There is another kind of automation that a database manager program can have. This is associating activity with specific events and data changes. Access does not provide this functionality, but SQL Server and Oracle do. You can associate a trigger with an event, such as changing a data value, and the software will run that action when that event happens. Entering and editing triggers can be done in one of two ways. The programs provide a way to create and modify triggers using the SQL Data Definition Language by entering commands interactively. They also provide a user interface for drilling down to the trigger as part of the table object model and entering and editing triggers. This interface for SQL Server is shown in Figure 40. A stored procedure is similar except that it is called explicitly rather than being associated with an event.
Calculated fields One of the things that computers do very well is perform calculations, and often retrieving data from a database involves a significant amount of this calculation. People who use spreadsheets are accustomed to storing formulas in cells, and then the spreadsheet displays the result. It is tempting to store calculated results in the database as well. In general, this is a bad idea (Harkins, 2001a), despite the fact that this is easy to do using the programmability of the database software. There are several reasons why this is bad. First, it violates good database design. In a well-designed database, changing one field in a table should have no effect on other fields in the table. If one field is calculated from one or more others, this will not be the case. The second and main reason is the risk of error due to redundant data storage. If you change one data element and forget to change the calculated data, the database will be inconsistent. Finally, there are lots of other ways to achieve the same thing. Usually the best way is to perform the calculation in the query that retrieves the data. Also, calculated controls can display the result on the fly. There are exceptions, of course. A data warehouse contains extracted and often calculated data for performance purposes. In deeply nested queries in Access, memory limitations sometimes required storing intermediate calculations in a table, and then performing more queries on the intermediate results. For the most part, however, if your design involves storing calculated results, you will want to take a hard look at whether this is the best way to proceed.
CHAPTER 7 THE USER INTERFACE
The user interface for the software defines the interaction between the user and the computer for the tasks to be performed by the software. Good user interface design requires that the software should be presented from the point of view of what the user wants to do rather than what the computer needs to know to do it. Also, with user interfaces, less is more. The more the software can figure out and do for the user without asking, the better. This section provides information that may be helpful in designing a good user interface, or in evaluating a user interface designed by others.
GENERAL USER INTERFACE ISSUES The user interface should be a modern graphical user interface with ample guidance for users to make decisions about the action required from them so that they can perform the tasks offered by the system. In addition, online help should be provided to assist them should they need additional information beyond what is presented on the software screens. The primary method for navigating through the system should be a menu, forms, or a combination of both. These should have options visible on the screen to take users to the areas of the program where they can perform the various activities. An example of a main menu form for an EDMS is shown in Figure 41. Users select actions by pressing labeled buttons on forms. Pressing a button will usually bring up another form for data input or viewing. Each form should have a button to bring them back to the previous form, so they can go back when they are finished or if they go to the wrong place. Forms for data entry and editing should have labeled fields, so it is clear from looking at the screen what information goes where. Data entry screens should have two levels of undo. Pressing the Escape key once will undo changes to the current field. Pressing it again will undo all changes to the current record. Multiple levels of undo can significantly increase the users’ confidence that they can recover from problems. The user interface should provide guidance to the users in two ways. The first is in the arrangement and labeling of controls on forms. Users should be able to look at a form and get enough visual clues so they know what to do. The second type of user interface guidance is tool tips. Tool tips are little windows that pop up when the cursor moves across a control providing the user with guidance on using that control. Consistency and clarity are critical in the user interface, especially for people who use the software on a part time or even an occasional basis.
86
Relational Management and Display of Site Environmental Data
The illusion of simplicity comes from focusing on only one variable. Rich (1996)
Figure 41 - Example of a menu form with navigation buttons
Figure 42 - Tool tip
CONCEPTUAL GUIDELINES An environmental data management program is intended to have a useful life of many years. During both the initial development and ongoing maintenance stages, it is likely that many different people will make additions and modifications to the software. This section is intended to provide guidance to those individuals so that the resulting user interface is as seamless as possible. The primary focus of these guidelines is on ease of use and consistency. These two factors, combined with a high level of functionality, will lead to a positive user experience with the software and acceptance of the system in the organization. A number of questions to be answered regarding user interface design are listed in Cooper (1995, p. 20). His advice for the answers to these questions can be broken down into two premises. The first is that the software should be presented from the point of view of what the user wants to do (manifest model) rather than what the computer needs to know to do it (implementation model). The second premise is that with user interfaces, less is more. The more the software can figure out and do for the user without asking, the better.
The User Interface
87
This section uses Cooper’s questions as a framework for discussing the user interface issues for an EDMS. The target data management system is a client-server system with Microsoft Access as the front end and SQL Server as the back-end, but the guidelines apply equally well for other designs. Answers are provided for a typical system design, but of course these answers will vary depending on the implementation details. It is important to note that the tools used in addressing user interface issues must be those of the system in which it is operating. Some of this material is based on interviews with users in the early stages of system implementation, and some on discussions with designers and experienced users, so it represents several perspectives. What should be the form of the program? – The data management system consists of tables, queries, forms, reports, macros, and modules in Microsoft Access and SQL Server to store and manipulate the data and present the user interface. For the most part, the user will see forms containing action buttons and other controls. The results of their actions will be presented as an Access form or report window. A recurring theme with users is that the system must be easy to learn and use if people are going to embrace it. This theme must be kept in mind during system design and maintenance. Every attempt should be made to ensure that the software provides users with the guidance that they need to make decisions and perform actions. The software should help them get their work done efficiently. How will the user interact with the program? – The primary interaction with the user is through screen forms with buttons for navigation and various data controls for selection and data entry. In general users should be able to make selections from drop-down lists rather than having to type in selections, and as much as possible the software should remember their answers to questions so that those answers can be suggested next time. The most common comment from users related to the user interface is that the system should be easy to use. People feel that they don't have much time to learn a new system. In order to gain acceptance, the new system will need to save time, or at least provide benefits that outweigh the costs of setup and use. Another way of saying this is that the software should be discoverable. The user should be able to obtain clues about what to do by looking at the screen. The example shown in Figure 43 shows a screen from a character-mode DOS interface. This interface is not discoverable. The user is expected to know what to type in to make the program do something. In this example, what the user tried didn’t work. The next example, Figure 44, shows a major improvement. Users need no a priori knowledge of what to do. They can look at the screen and figure out what to do by reading their options. Of course, even a good idea can have flaws. In this example, the flow is a little illogical, expecting users to click on Start to stop (shut down) their computer, but the general idea is a great improvement. The transition to a discoverable interface, especially at the operating system level, which was originally popularized by the Apple Macintosh computer and later by the Microsoft Windows operating system, has made computer use accessible to a much wider audience. How can the program’s function be most effectively organized? – The functions of the program are organized by the tasks to be performed. In most cases, users will start their session by selecting a project, and perhaps other selection criteria, and then select an action to perform on the selected data. Where a set of answers is required, the questions they are asked should be presented in a clear, logical sequence, either on a single form or as a series of related “wizard”-like screens. How will the program introduce itself to first-time users? – An example of program introduction would be for the program to display an introductory (splash) screen followed by the main menu. An on-screen “tour” or tutorial screen as shown in Figure 45 can be very helpful in getting a new user up and running fast.
88
Relational Management and Display of Site Environmental Data
Figure 43 - Example of an interface that is not “discoverable”
Figure 44 - Example of a “discoverable” interface
Figure 45 - On-screen “tour” or tutorial
A printed tutorial can perform this function also, but experience has shown that people are more likely to go through the tutorial if it is presented on-screen. This satisfies the “instant gratification” requirement of someone who has just acquired a new software program and wants to see it do something right away (the “out of box” experience). Then after that users can take the time to learn the program in detail, get their data loaded, and use the software to perform useful work. How can the program put an understandable and controllable face on technology? – The software must make the user comfortable in working with the data. The user interface must make it easy to experiment with different approaches to retrieving and displaying data. This allows people to find the data selection and presentation approach that helps them best communicate the desired message. Figures 46 and 47 provide interesting examples of the good and the bad of using multiple display windows to help the user feel comfortable working with data. The multiple windows showing different ways of looking at the data would confuse some users. Others would be thrilled to be able to look at their data in all of these different ways. There is certainly a personal preference issue regarding how the software presents data and lets the user work with it. The software should provide the option of multiple on-screen displays, and users can open as many as they are comfortable with.
The User Interface
Figure 46 - Database software display with many data elements
Figure 47 - GIS software display with many data elements
89
90
Relational Management and Display of Site Environmental Data
Several things can be done in the user interface to support these usability objectives. As discussed above, the software should be discoverable. It should also be recoverable, so that users can back out of any selection or edit that they may make incorrectly. Users should be provided with information so that they can predict the consequences of their actions. For example, the software should give clues about how long a process will take, and the progress of execution. Any process that will take a long time should have a cancel button or other way to terminate processing. This goal can sometimes be difficult to accomplish, but it’s almost always worth the effort. How can the program deal with problems? – When an error occurs, the program should trap the error. If possible the software should deal with the error without user intervention. If this is not possible, then the program should present a description of the error to the user, along with options for recovery. The software designers should try to anticipate error conditions and prepare the software and the users to handle them. How will the program help infrequent users become more expert? – The user should be able to determine how to perform their desired action by looking at the screen. Tool tips should be provided for all controls to assist in learning, and context-sensitive help at the form level should be provided to make more information available should the user require it. How can the program provide sufficient depth for expert users? – The forms-based menu system should provide users with the bulk of the functionality needed to perform their work. Those wishing to go beyond this can be trained to use the object creation capabilities of the EDMS to make their own queries, forms, and reports for their specific needs. This is a major benefit of using a database program like Access rather than a compiled language like Visual Basic to build the EDMS front-end. Training is important at this stage so users can be steered away from pitfalls that can be detrimental to the quality of their output. For example, a section in Chapter 15 discusses data retrieval and output quality issues. Another way of looking at this issue is that the program should be capable of growing with the user.
GUIDELINES FOR SPECIFIC ELEMENTS Automated import, data review, manual data entry, lookup table maintenance, and related administrative activities should be done in forms, queries, and modules called up from menu choices. The rest of the interaction with the system is normally done through the selection screen. This screen is a standardized way for the user to select a subset of the data to work with. They should be able to select a subset based on a variety of data elements. The selection process starts with a base query for each of the data elements. The selection screen then appends to the SQL “WHERE” clause based on the items they have selected on the screen. This query is then saved, and can be used as the basis for further processing such as retrieving or editing data. The system should make it easy, or at least possible, to add new functions to the system as specific needs are identified. The user interface is the component of the EDMS that interacts with the people using the software. The user interface for a data management system consists of five principal parts: Input, Editing, Selection, Output, and Maintenance. Input – This section of the user interface allows data to be put into the system. In an EDMS, this involves file import and manual input. File import allows data in one or more specified formats to be brought into the system. The user interface component of file import should let the user select the location and name of the file, along with the format of the file being imported. They should be able to specify how various import options like parameter and unit conversion will be handled. Manual input allows data to be typed into the system. The procedures for file import and manual input must provide the necessary level of quality assurance before the data is considered ready for use. Editing – It is necessary to provide a way to change data in the system. The user interface component of data editing consists of presenting the various data components and allowing them to
The User Interface
91
be changed. It is also critical that the process for changing data be highly controlled to prevent accidental or intentional corruption of the data. Data editing procedures should provide a process to assure that the changes are valid. Components of the data management software, such as referential integrity, lookup tables, and selections from drop-down menus, can help with this. Selection – The two most important parts of an EDMS are getting the data in and getting the data out. Especially in larger organizations, getting the data in is done by data administrators who have been trained and have experience in carefully putting data into the system. Getting the data out, however, is often done by project personnel who may not be as computer literate, or at least not database experts. The user interface must address the fact that there will be a range of types of users. At one extreme is the type of user who is not familiar or comfortable with computers, and may never be. In the middle are people who may not have had much experience with data management, but will learn more over time. At the high extreme are power users who know a lot about data management coming in and want to roll their sleeves up and dig in. The software should make all of these types of users comfortable in selecting and outputting data. A query by form (QBF) selection screen is one way to accomplish this. Output – Once a subset of the data has been selected, the software should allow the data to be output in a variety of formats. For a system like this, almost all of the output follows a selection step. The selection for output involves choosing the data content (tables and fields), record subset (certain sites, stations, sample dates, etc.) and output format. The software should provide a set of standard (canned) output formats that can be chosen from the QBF selection screen. These can range from relatively unformatted lists to formalized reports, along with graphs and maps, and perhaps output to a file for further processing. Maintenance – All databases require maintenance, and the larger the database (number of records, number of users, etc.), the more maintenance is required. For an EDMS, the maintenance involves the server and the clients. The largest item requiring maintenance in the server component of an EDMS is data backup, which is discussed in more detail in Chapter 15. Another server task is maintenance of the list of users and passwords. This must be done whenever people or assignments change. Also, the data volumes in the database manager may need to be changed occasionally as the amount of data increases. This is usually done by a computer professional in IS. Some maintenance of the client component of the EDMS is usually required. Access databases (.mdb files) grow over time because temporary objects such as queries are not automatically removed. Consequently, maintenance (compacting) of the .mdb files must be performed on an occasional basis, which can vary from weekly to monthly depending on the level of use. Also, occasionally .mdb files become corrupted and need to be repaired. This can be automated as well. Finally, as improvements are made to the EDMS system, new versions of the program file containing the front end will need to be distributed to users, and a simple process should be developed to perform this distribution to a large number of users with a minimum of effort.
DOCUMENTATION The three main factors that lead to a satisfactory user experience with software are an intuitive user interface, accessible and clear documentation, and effective training. The data management system should have two primary documentation types, hard copy and online. The online documentation consists of two parts, the user interface and the help file. The hard copy documentation and help file will be described in the following sections.
Hard copy The hard copy documentation should consist of two parts, a tutorial section and a reference section. The tutorial section should take the user through the process of using the software and
92
Relational Management and Display of Site Environmental Data
working with data. The reference section should cover the various aspects of working with the system, and have a goal of anticipating and answering any questions the user might have in working with the software. Both sections should have appropriate illustrations to help the user understand what is being described. In addition to covering the day-to-day operation of the system, the documentation should also cover maintenance procedures for both the client and server sides of the system.
Help file The data management system should be installed with a complete online help file using the standard Windows Help System. It can be based on the hard copy documentation, with any modifications necessary to adapt that documentation to the online format. Help screens should be provided for each form in the system (form-level context-sensitivity).
CHAPTER 8 IMPLEMENTING THE DATABASE SYSTEM
This chapter addresses the process of getting a database management system up and running. Topics covered include designing the system, installing the system, and other issues related to the transition from design and construction to ongoing use.
DESIGNING THE SYSTEM The design of the database system is important if the goals of the system are to be satisfied. This section covers a number of issues related to designing the system.
General design goals Before designing a database system, the goals and needs for the system should be clearly identified. To design a usable system, both general design goals such as those described in the literature and goals specific to the organization must be taken into account. This section presents the design goals for an EDMS from a general perspective. The objectives of database design are generally the same for all organizations and all database systems. Stating these goals can help with implementation decisions because the choice that furthers the greatest number of goals is usually the right choice. Some of the material in this section is discussed in Jennings (1995, pp. 828-829) and Yourdon (1996). The most important aspect of designing a database management system is determining the business problem to be solved. Once this has been done, many of the design decisions are much easier to make. In designing and implementing this system, the following data management goals should be considered: Organization goals that should be addressed include: • • •
Fulfilling the needs of the organization for information in a timely, consistent, and economical manner. Working within the hardware and software standards of the organization, and, as much as is practical, using existing resources to minimize cost. Accommodating expansion of the database to adapt to changing organizational needs.
94
Relational Management and Display of Site Environmental Data
Planning for flexibility must be a part of the system design. Business rules have longevity, but processes change. The technology that models the business process must be able to change in order to have any longevity. Data management goals that should be considered include: • • • •
Providing rapid access to data required by each user category. This includes providing a user interface that is easy to learn and easy to use, yet flexible and powerful enough to allow any kind of data retrieval. Eliminating or minimizing the duplication of data across the organization. Easing the creation or integration of data entry, data review, editing, display, and reporting applications that efficiently serve the needs of the users of the database. Preserving data for future use. Any significant data available in digital form should be stored in the database, and backup and recovery procedures should be in place to preserve the content.
Database security goals include: • • • • •
Maintaining the integrity of the database so that it contains only validated, auditable information. Preventing access to the database by unauthorized people. Permitting access only to those elements of the database information that individual users or categories of users need in the course of their work. Allowing only authorized people to add or edit information in the database. Tracking modifications to the data.
Quality goals for the system could be: • • •
Designation of responsibility for decisions about data included in the database. Designation of responsibilities for decisions about data gathering. Use of approved data collection procedures.
Database project management issues include: • • • • • • • • • •
Responsibilities for data management should be clearly defined. The system should provide the container for data. However, project managers should decide how to use it based on the business model for their project, since the level of detail that is appropriate may vary from project to project. Potential uses for the data should be identified so that the quality of the data gathered will match the intended use for that data. Objectives of the data gathering effort should be clearly and unambiguously established. Where several organizations are separately collecting data for multimedia assessments, it is essential that efforts be made to identify and discuss the needs of the principal users of the data. This includes establishment of minimum data accuracy. Once the intended uses for the data are defined, then a quality control program should be designed and implemented. Data needs should be periodically reviewed in the light of increased understanding of environmental processes, or changes in toxicological, climatic, or other environmental conditions. To get the full use of a database system, correct data collection procedures should be used to achieve the highest possible quality for the data being entered into the database. Accepted measurement and data handling methodologies should be used whenever possible. Well-tested measurement practices and standard reference materials should be developed if not already in use. This will allow adequate quality control practices to be implemented in measurement programs.
Implementing the Database System • • • •
95
Protocols for data measurement and management should be periodically reviewed. The quality of the collected data should be carefully reviewed and documented before the data is made available for general use. When data-gathering procedures are changed, care should be taken to assure that the old data can be correlated with the new set with no loss in continuity. Information on existing data programs and data and measurement standards should be disseminated widely to the data user community.
Determine and satisfy needs It is possible to develop a standard procedure for completing the project on time and on budget. This procedure has several steps, which will be described here. Many of these steps are discussed in more detail in later sections, but are included here to make sure that they are considered in the planning process. Assess the needs of the organization – This is probably the most important step in the process. This process should be continued on into implementation. One good process is to select a cross section of potential users and other interested parties within the organization and interview them from a prepared questionnaire. (See Appendix A.) This questionnaire should be prepared for each project based on a template from previous similar projects. The questions on the form progress from general to specific in order to elicit each user’s needs and interests for the system. An important factor in selecting technology is the attitudes of your organization and the individuals in it toward the adoption of new technology. Moore (1991), in his popular book Crossing the Chasm, describes the Technology Adoption Life Cycle, and groups people by their response to technology. Gilbert (1999) has related these groups to software implementation in environmental organizations. These groups are Innovators, Early adopters, Early majority, Late majority, and Laggards. The chasm is between Innovators and Early adopters, and it is difficult to move technology across this chasm. This concept is important in selling and then managing a technology project in your organization. You should analyze the decision makers and prospective users of the technology, and choose technology at the appropriate level of innovation that will be comfortable (or at least not too uncomfortable) for them. Create a plan – Based on the results of the questionnaire, the implementation team should work with a small group of individuals within the target user group to develop a data management plan. This plan serves several major purposes. It provides a road map for the long-term direction of the project. It serves as a design document for the initial release of the system. And it helps facilitate discussion of the various issues to be addressed in developing and implementing the system. This often-overlooked step is critical to project success. Fail to plan, plan to fail. Develop the data model – This is also a critical step in the process. In this step, which can be done parallel with or subsequent to creation of the plan, the implementation team and the users work to make sure that the data content of the system meets the needs of the users. Perform software modifications – If the needs assessment or the data model design identifies changes which need to be made in the software, they are performed in this step. If the database software is based on an open system using standard tools, these changes are usually quite straightforward. Of course, the level of effort required by this step is proportional to the number and scope of the changes to be made. A key action that must be carried out throughout all five of the above steps is to communicate with the future users. Too often, the team writing the software creates what they think the users want, only to find out weeks or months later that the users’ needs are totally different. Frequent communication between the developers and users can help prevent frustration and project failure. Test, then test again – Once the data model and software functionality have been implemented, the system must be fully tested. The installation team should test the software during and after the modifications and remedy any difficulties encountered at that time. Once the team
96
Relational Management and Display of Site Environmental Data
members are satisfied that they have found all of the problems that they can, the software must be tested in the different and often more varied environment of the client site. It is better to start out with a small group of knowledgeable users, and then expand the user base as the number of problems encountered per unit of use time decreases. When the client and the implementation team agree that the problem rate is at an acceptable level, the software can be released for general use. Document – Good documentation is important for a successful user experience. Some users prefer and will read a written manual. Others prefer online help. Both must be provided. Train – Most users learn best from a combination of formally presented material and hands-on use. It is useful to have materials to teach classes for a variety of different types of users, and these materials can be modified prior to presentation to reflect the anticipated use pattern of the target users. A facility suitable for hands-on training must be provided that is in a location that is convenient for the students. Support – When the user has a problem, which they will despite the best developing and testing efforts, there must be a mechanism in place for them to get help. The actual execution of these steps varies from client to client, but by following this process, the project has the greatest chance for success.
Prepare a plan A data management plan is intended to provide guidance during the detailed design and implementation of an EDMS. Design of a computerized data management system should begin with a survey of the data management needs of the organization. It should integrate knowledge about the group’s information management needs gathered during the needs assessment phase, together with the necessary hardware and software components to satisfy as many of the data management needs as possible. It is intended that this plan be revised on a regular basis, perhaps semi-annually, as long as the data management system is in use. The expected life of a typical data management system is five to ten years. After that period, it can be anticipated that technology will have changed sufficiently and it will be appropriate to replace the system with a new tool. It is reasonable to expect that the data from the old system can be transported into the new system. Even during the life of the software it will be necessary to make changes to the system to accommodate changes in computer technology and changes in data needs. This is particularly true now, as the computer industry is undergoing a change from traditional client-server systems to browser-based Internet and intranet systems. Since the data management plan often involves an incremental development process where functionality is added over time, you should expect that lessons learned from early deployment will be incorporated into later development. Finally, the level of detail provided in the plan may vary in the discussion of the different data types to be stored. Additional detail will be added as software development progresses. You should allow the plan to evolve to address these and other, perhaps unanticipated, system changes. A typical data management plan might contain the following sections: Section 1 – System Requirements Section 2 – System Design Section 3 – Implementation Plan Section 4 – Resource Requirements Appendix A – Database Fundamentals Appendix B – User Interface Guidelines Appendix C – Preliminary Data Model Appendix D – Preliminary System Diagrams Appendix E – Data Transfer Standard Appendix F – Coded Values
Implementing the Database System
97
Appendix G – Data Management Survey Results Appendix H – Other Issues and Enhancements Appendix I – References The plan should contain a discussion of all of the important issues. The level of detail may vary depending on the particular item and its urgency. For example, at the planning stage the data content of the system may be outlined in broad terms. After identification of the data components that are the most significant to potential users the data content can be filled in with more detail. The next step toward implementation of the database management system is the detailed design.
Design the system in detail The next step after finalizing the data management plan will usually be a detailed system design. This detailed design will identify and respond in detail to various issues related to each data type being addressed. The plan should be viewed as a national road map. The national map provides an overview of the whole country, showing the relationships between different areas and the high-level connections between them, such as the Interstate highways. It contains greater detail in some areas, just as a national map may have more detailed sub-maps of some metropolitan areas. These sub-maps contain adequate detail to take you wherever you want to go in those areas. The detailed system design covers this greater detail. In many cases, the detailed design is not prepared entirely in one step, but parts of the system are designed in detail prior to implementation, in an evolving process. Figure 48 shows an example of this iterative process. This example shows a preliminary data model that was designed and then submitted to prospective users for comment. The result of the meeting to discuss the data model was the notes on the original data model shown in the figure. After several sessions of this type, the final data model was completed, which is shown at a reduced scale in Figure 49. This figure illustrates the complexity that resulted from the feedback process. It’s important to catch as many errors as possible in this stage of the design process. Conventional wisdom in the software development business is that the cost to fix an error that is found at the end of the implementation process is a factor of 80 to 100 greater than to fix the same error early in the design process. It’s worthwhile to beat the design process to death before you proceed with development, despite the natural desire to move on. One or two extra design sessions after all involved think they are happy with the design will usually pay for themselves many times over.
BUY OR BUILD? After the needs have been determined and a system design developed, you will need to decide whether to buy existing software or write your own (or have it written for you). A number of factors enter into this decision. The biggest, of course, is whether there is software that you can buy that does what you want. The closer that existing software functionality matches your needs, the easier the decision is. Usually, it is more cost-effective when you can buy rather than build, for several reasons, mostly related to the number of features relative to the cost of acquiring the system. There is a cultural component to the decision, with some organizations preferring to write software, while others, perhaps with less interest or confidence in their development capabilities, opting to buy when possible. There may be times when you have to bite the bullet and write software when there is no viable alternative, and the benefits justify the cost.
Relational Management and Display of Site Environmental Data
Figure 48 - Intermediate step in the detailed design process
98
Figure 48 - Intermediate step in the detailed design process
Implementing the Database System
99
Confidence is the feeling you have before you understand the situation. Rich (1996)
Figure 49 - Reduced data model illustrating the complexity resulting from the detailed design process in the previous figure
There is a definite trend in the environmental consulting business away from custom database software development. This is due to two primary reasons. The first is that commercial software has improved to the point that it often satisfies all or most project needs out of the box. The second is more scrutiny of project budgets, with clients less willing to pay for software development unless it is absolutely necessary. Abbott (2001) has stated that 31% of software development projects are canceled before they are completed, and 53% ultimately cost 189% or more of their original budgets. Software projects completed by large companies typically retain about 42% of the features originally proposed. Buying off-the-shelf software decreases the chance of one of these types of project failure. It would be helpful to be able to estimate the cost of implementing a database system. Vizard (2001) states that $7 of every $10 spent on software goes into installing and integrating the software once it is purchased. Turning this around, for every dollar spent on purchasing or developing the software, about two more are spent getting it up and running. So for a rough estimate of the project cost, take the cost of the software and triple it to get the total cost of the implementation.
IMPLEMENTING THE SYSTEM Once the system is selected or designed, there are a number of tasks that must be completed to get it up and running. These same basic tasks are the same, whether you are buying or building the software.
Acquire and install the hardware and software The process of selecting, purchasing, and installing a data management system, or of writing one, can be quite involved. You should be sure that the software selected or developed fits the needs of the individuals and the organization, and that it will run on existing hardware. It may be necessary to negotiate license and support agreements with the vendor. Then it will be necessary to install the software on users’ computers and perhaps on one or more servers.
100
Relational Management and Display of Site Environmental Data
Special considerations for developing software If you are building the software instead of buying it, it is important to follow good software development practices. Writing quality software is a difficult, error-prone process. For an overview of software quality assurance, see Wallace (2000). Here are a few tips that may help: Start with a requirements plan – Developing good software starts with carefully identifying the requirements for the finished system as described above (Abbott, 2001). According to Abbott, 40 to 60% of software defects and failures result from bad requirements. Getting everyone involved in the project to focus on developing requirements, and getting developers to follow them, can be very difficult, but the result is definitely worthwhile. Abbott quotes statistics that changes that occur in the development stage cost five times as much as those that occur during requirements development, and once the product is in production, the cost impact is a factor of 100. Use the best tool for the job – Choose the development environment so that it fits project needs. If code security is important, a compiled language such as Visual Basic may be a good choice. If flexibility and ease of change is important, a database language like Access is best. Manage change during development – Even with the best plan, changes will occur during development, and managing these changes is critical. On all but the smallest projects, a formal change order process should be used, where all changes are documented in writing, and all stakeholders sign off on every change before the developer changes code. A good guideline is ANVO (accept no verbal orders). Use prototypes and incremental versions – Developers should provide work examples to users early and throughout the process, and solicit feedback on a regular basis, to identify problems as early as possible in the development process. Then the change order process can be used to implement modifications. Manage the source code – Use source code management software or a formal process for source code management to minimize the chance for conflicting development activities and lost work, especially on projects with multiple developers. Implement a quality program – There are many different types of quality programs for software development. ISO 9000 for quality management and ISO 14000 for environmental management can be applied to EDMS software development and use. TQM, QFD, SQFD, and CMM are other examples of quality programs that can be used for any software development. Which program you choose is less important than that you choose one and stick to it. Write modular code and reuse when possible – Writing software in small pieces, testing these pieces thoroughly, and then assembling the pieces with further testing is a good way to build reliable code. Where possible centralize calculations in functions rather than spreading them across forms, reports, and code. Each module should have one entry and one exit point. Use syntax checking – Take advantage of the syntax checking of your development environment. Most languages provide a way for the program to make sure your syntax is valid before you move on, either by displaying a dialog box, or underlining the text, or both. Format your code – Use indentation, blank lines, and any other formatting tools you can think of to make your code easier to read. Explicitly declare variables – Always require that variables be explicitly declared in the declarations section of the code. In Visual Basic and VBA you can require this with the statement Option Explicit in the top of each module, and you can have this automatically entered by turning on Require Variable Declaration in the program options. Watch variable scope – The scope of variables (where in the code they are active) can be tricky. Where possible declare the scope explicitly, and avoid re-using variable names in subroutines in case you get it wrong. Keeping the scope as local as possible is usually a good idea. Be very careful about modifying global variables within a function. Design tables carefully – Every primary table should have a unique, system-assigned ID and an updated date to track changes.
Implementing the Database System
101
Be careful about error handling – This may be the one item that separates amateur code from professional code. A good programmer will anticipate error conditions that could arise and provide error trapping that helps the user understand what the problem is. For example, if the detection limit is needed for a calculation but is not present, an EDMS error message like “Please enter the detection limit” is much more helpful than a system message like “Invalid use of Null.” Use consistent names – The more consistent you are in naming variables and other objects the easier it will be to develop and support the code. There are several naming conventions to choose from, and how you do it is not as important as doing it consistently. An example of one such system can be found in Microsoft (2001). For example, it is a good idea to name variables based on their data type and the data they contain, such as txtStationName for the text field that contains station names. Avoid field and variable names that are keywords in the development language or any other place that they may be used. Document your code – Document code internally and provide programmer documentation for the finished product. Each procedure should start with a comment that describes what the function does, as well as its input and output variables. Inline comments in the code should be plentiful and clear. Don’t assume that it will be obvious what you are doing (or more importantly why you are doing it this way instead of some other way), especially when you try to get clever. Don’t get out of your depth – Many advanced computer users have an exaggerated view of their programming skills. If you are building a system that has a lot riding on it, be sure the person doing the development is up to the task. Don’t forget to communicate – On a regular basis, such as weekly or monthly, depending on the length of the project, or after each new section of the system is completed, talk to the users. Ask them if the direction you are taking is what they need. If yes, proceed ahead. If not, stop work, talk about directions, expectations, solutions, and so on, before you write one more line of code. This could make the difference between project success and failure.
Test The system must be thoroughly tested prior to release to users. This is usually done in two parts, alpha testing and beta testing, and both are quite valuable. Each type of testing should have a test plan, and should be carefully documented and methodically followed. Alpha testing – Alpha testing is testing of the software by the software developer after it has been written, but before it is delivered to users. Alpha testing is usually performed incrementally during development, and then comprehensively immediately before the software is released for beta testing. For commercial software purchased off the shelf, this should already have been done before the software is released to the general public. For custom software being made to order, this stage is very important and, unfortunately, in the heat of project deadlines, may not get the attention it deserves. The test plan for alpha testing should exercise all features of the software, and when a change is made, all previous tests should be re-run (regression testing). Test items should include logic tests, coverage (the software works for all cases), boundary tests (if appropriate), satisfaction of requirements, inputs vs. outputs, and user interface performance. It is also important to test the program in all of the environments where the software is going to be deployed. If some users will be using Access 97 running under Windows 95, and others Access 2000 and Windows 2000, then a test environment should be set up for each one. A program like Norton Ghost, which lets you easily restore various hard drive images, can be a great time saver in setting up test systems. A test machine with removable drives for different operating system versions can be helpful as well, and is not expensive to set up. Don’t assume that if a feature does what you want in one environment that it will work the same (or at all) in a different one. Beta testing – After the functionality is implemented and tested by the software author, the software should be provided to willing users and tested on selected computers. There is nothing
102
Relational Management and Display of Site Environmental Data
like real users and real data to expose the flaws in the software. The feedback from the beta testers should then be used to improve the software, and the process repeated as necessary until the list of problem reports each time is very short, ideally zero. The test plan for beta testing is usually less formal than that for alpha testing. Beta testers should use the software in a way similar to how they would use the final product. A caveat here is that since the software is not yet fully certified it is likely to contain bugs. In some cases these bugs can result in lost or corrupted data. Beta testers should be very careful about how they use the results obtained by the software. It is best to consider beta testing as a separate process from their ordinary workflow, especially early in the beta test cycle.
Document An important part of a successful user experience is for users to be able to figure out how to get the software to do the things that they need to do. Making the software discoverable, so that visual clues on the screen help them figure out what to do, can help with this. Unfortunately this is not always possible, so users should be provided with documentation to help them over the rough spots. Often a combination of printed and online documentation provides the best source of information for the user in need. The response of users to documentation varies tremendously from user to user. Some people, when faced with a new software product, will take the book home and read it before they start using the program. Other users (perhaps most) will refuse to read the manual and will call technical support with questions that are clearly answered in both the manual and the help file. This is a personality issue, and both types of help must be provided.
Train Environmental professionals are usually already very busy with their existing workloads. The prospect of finding time to learn new software is daunting to many of these people. The implementation phase must provide adequate training on use of the system, while not requiring a great time commitment from those learning it. These somewhat contradictory goals must be accommodated in order for the system to be accepted by users. New training tools on the horizon, such as Web-based, self-based training, show promise in helping people learn with a minimum impact on their time and the organization’s budget. Training should be provided for three classes of users. System administrators manage the system itself. Data administrators are responsible for the data in the system. Users access the data to assist them in their work. Usually the training of administrators involves the greatest time commitment, since they require the broadest knowledge of the system, but since it is likely to be an important part of their job, they are usually willing to take the time to learn the software. System administrators – Training for System administrators usually includes an overview of the database tools such as Access, SQL Server, or Oracle; the implementation of the data management system using these tools; and operation of the system. It should cover installation of the client software, and maintenance of the server system including volume management, user administration, and backup and restoration of data. Data administrators – Training for data administrators should cover primarily data import and editing, including management of the quality tracking system. Particular emphasis should be placed on building a thorough understanding of the data and how it is reported by the laboratories, since this is a place where a lot of the problems can occur