php|architect (June 2005)

”; } // Close database delete_writabledatabase ($db); ?>

June 2005

●

PHP Architect

●

www.phparch.com

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

“To be or not to be”, “Spartacus” => “I am Spartacus”, “Star Wars” => “Luke, I am your father”); dl (“xapian.so”); $db = new_database (“db”); if (!isset ($_POST[‘search’]) || $_POST[‘search’] === ‘’) { die (“Please specify a search string.”); } $search_terms = str_word_count ($_POST[‘search’], 1); foreach ($search_terms as $v) { $new_query = new_query (strtolower ($v)); if (!$query) { $query = $new_query; } else { $query = new_query_from_query_pair (Query_OP_OR, $query, $new_query); } } echo “Searching for “. urlencode($_POST[‘search’]) .”

”; $enq = new_enquire ($db); enquire_set_query ($enq, $query); $result_set = enquire_get_mset ($enq, 0, 10); echo mset_get_matches_estimated($result_set) . “ results found

”; $item = mset_begin ($result_set); while ( !msetiterator_equals($item, mset_end($result_set)) ) { echo htmlentities( $search_strings[document_get_data( msetiterator_get_document($item))]) . “
[ID: “ . msetiterator_get_docid ($item) . “ - Data: “ . htmlentities (document_get_data ( msetiterator_get_document ($item))) . “ - Match: “ . msetiterator_get_percent ($item) . “]
”; msetiterator_next($item); } ?>

49

Homo Xapian: A Search for a Better Search...Engine

throughout, so once you’ve figured out how things work, you shouldn’t have any further problems. Xapian, Quartz and Databases Xapian is built to be capable of relying on a number of different database backends that provide the necessary data storage and retrieval mechanisms for the search engine to function. The reason for this is that different applications have different needs—for example, an inmemory database backend provides maximum performance at the cost of memory usage and the inability to permanently save the database. For the most part, however, you will find yourself

Listing 3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

“string2” => “string3” =>

= array “He kept rolling down the stairs”, “Living la vida loca”, “Click the cat has seven lives”);

// The following adds a document to the database. // Note how each word is converted to lowercase before // being added as a search term. function add_document ($db, $doc_data, $key) { global $stemmer; $words = str_word_count ($doc_data, DB_CREATE_OR_OPEN); // Create a new document $doc = new_document(); // Set the key as our metadata document_set_data ($doc, $key); echo “Indexing ‘$doc_data’ with key ‘$key’
”; // Index each word separately // but specifying the order in which foreach ($words as $k => $v) { document_add_posting ($doc, stem_stem_word ($stemmer, strtolower ($v)), $k + 1); } // Add the document and return the ID return writabledatabase_add_document ($db, $doc); } // Load the Xapian extension dl (‘xapian.so’); // Open database $db = new_WritableDatabase (“db”, DB_CREATE_OR_OPEN); // Create stemmer $stemmer = new_stem (“english”); // Index the documents foreach ($search_strings as $k => $v) { $id = add_document ($db, $v, $k); echo “Document has ID #$id

”; } // Close database delete_writabledatabase ($db); ?>

June 2005

●

PHP Architect

●

www.phparch.com

FEATURE using Quartz, Xapian’s main database engine. Quartz offers a very robust data storage system that is highly scalable and supports reader/writer concurrency— meaning that you can update the database while an arbitrary number of search queries are being executed (although, you can’t have two processes update it at the same time). Regardless of what database backend you choose, there is one bit of good news: you won’t have mess around with anything even remotely approaching the SQL language. In fact, you will only have to interact with the database through Xapian methods, which, if you’ve ever had to insert data from a fifty-field form into a SQL database, can only be an improvement. Writing the Indexer In the Xapian model, a search engine is composed of two different entities: an indexer and a searcher. As its name implies, the former’s role is that of adding documents to the search engine’s database so that queries can later be run against them. Before being able to add data to a database, we have to ensure that we actually have a database. This is accomplished by initializing one—or simply opening if it already exists—through a call to new_WritableDatabase(). This function takes only one parameter, which is the name of the directory in which the database files are to be stored. Clearly, this directory should be in a location that is not directly accessible from the web, but to which your webserver and indexer’s processes both have the appropriate access rights. Once the database has been opened, we can move on to actually creating entries in it. Xapian defines a document as the smallest possible indexable entity. Note that “document” doesn’t necessarily mean a text file—you could be indexing all sorts of data, even of a non-textual nature, if you wanted to. The database engine assigns each document a unique ID, while we are allowed to store an arbitrary amount of data in it that will be returned to us when the document is retrieved by a query against the database. This will allow us to rapidly retrieve the resource associated with it by providing a token that can be used to rapidly identify an external entity—for example, if we are indexing PDF files, we could save the file path, while if we are crawling a remote website, we can store the URL of each page. It is not a good idea to store the actual resources in the database for a number of reasons, chiefly the fact that the Xapian database was designed as a repository of search metadata—and not as a general-purpose database. Keeping your documents in it will serve no purpose other than bloating its files, since you can’t perform any operation on the data—even retrieving it—without going through a search query. Finally, each document contains an arbitrary number of search terms, which will be used by the search engine

50

FEATURE

Homo Xapian: A Search for a Better Search...Engine

to match results against a query. It’s important to understand that search terms are used by the search engine without applying any semantic rules to them. This gives Xapian maximum flexibility by allowing you to use any sort of data—including binary data—as a search term, but it does bring up one interesting problem that you must be aware of when dealing with text. Since no semantic rules are attached to the search terms, textual data will be treated literally and in a casesensitive way by the engine. This means that “Marco” will be different from “marco,” and “search” will be different from “searching.” Thus, without proper intervention, arbitrary search queries from an end user are likely to fare rather poorly. Luckily, it’s very easy to solve this problem, as we’ll see in a moment. A new document is created by calling the new_document() function. We can then add data and search terms to it by calling document_set_data() and document_add_posting() respectively. Finally, we can insert the document into the database by calling WritableDatabase_add_document(). This can be optionally followed by a call to WritableDatabase_flush(), which will cause the document to be written to disk and made immediately available for searching. This simple process can be repeated as many times as needed—you simply create a new document for every item that needs to be indexed. Once you’ve processed of all your documents, you need to call delete_WritableDatabase() in order to remove your exclusive-write lock on the database; failing to do so will result in the database to become inaccessible for writing. In case of a crash, removing the lock is simple enough—all you have to do is delete a single file in your database directory—but an automated process that doesn’t delete its locks will effectively stop working until a system administrator can manually intervene. Listing 1 shows a very simple indexer. We start (on line 42) by loading the Xapian extension, which, if you compiled it properly, will already be stored in the default extension location for your PHP installation. We then follow the procedure that I outlined earlier in this section. Note how each word is converted to lowercase before being added to the appropriate document as a search term. Also, I used the str_word_count() function to break down each source string into its component words, and ensured that the script will add the latter to its document in the same order in which they appear in the former. This will make it possible for the search engine to calculate a proper proximity map when queries are executed later on. Building the Searcher It’s now time to move on to the portion of our application that actually performs the search. The procedure that we must follow here is extremely simple—first, we

June 2005

●

PHP Architect

●

www.phparch.com

Figure 1

Listing 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

“string2” => “string3” =>

array “He kept rolling down the stairs”, “Living la vida loca”, “Click the cat has seven lives”);

dl (“xapian.so”); $db = new_database (“db”); $stemmer = new_stem (“english”); if (!isset ($_POST[‘search’]) || $_POST[‘search’] === ‘’) { die (“Please specify a search string.”); } $search_terms = str_word_count ($_POST[‘search’], 1); foreach ($search_terms as $v) { $new_query = new_query ( tem_stem_word ($stemmer, strtolower ($v))); if (!$query) { $query = $new_query; } else { $query = new_query_from_query_pair (Query_OP_OR, $query, $new_query); } } echo “Searching for “. htmlentities ($_POST[‘search’]) . “

”; $enq = new_enquire ($db); enquire_set_query ($enq, $query); $result_set = enquire_get_mset ($enq, 0, 10); echo mset_get_matches_estimated($result_set) . “ results found

”; $item = mset_begin ($result_set); while (! msetiterator_equals ($item, mset_end ($result_set))) { echo htmlentities ($search_strings[document_get_data ( msetiterator_get_document ($item))]) . “
[ID: “ . msetiterator_get_docid ($item) . “ - Data: “ . htmlentities (document_get_data ( msetiterator_get_document ($item))) . “ - Match: “ . msetiterator_get_percent ($item) . “]
”; msetiterator_next($item); } ?>

51

FEATURE

Homo Xapian: A Search for a Better Search...Engine

Figure 2

Figure 3

must open our database in read-only more (remember, there can only be one writer at any given time, but the system supports an arbitrary number of concurrent readers). Next, we build a query based on user input and execute it against the database. Finally, we retrieve the appropriate result set from the latter and ask the search engine to also return an estimate of the total number of hits. Building the query is definitely the most complicated task at hand. A query is composed of a set of search terms joined together by Boolean operators. Sounds complicated? It can be. Consider, for example, the following search string: “Marco Tabini” PHP

If we typed this string in a search engine like Google, we’d expect it to execute a search for the string “Marco Tabini” in proximity of the word “PHP.” In reality, what we are telling the search engine is that we want to execute a query like the following: (MARCO and TABINI) or PHP

You probably weren’t expecting the or operator right after the closed parenthesis—but, as far as Xapian is concerned, its use only causes the search engine to perform a probabilistic match, which is what we are after. Once you have determined how your search terms should be joined together, you can create a new query by using the new_Query() function. In this context, a query is simply the search for a given search term—for example, the search string above contains three different queries: one for the word “MARCO,” one for “TABI-

June 2005

●

PHP Architect

●

www.phparch.com

NI” and one for “PHP.” Additional queries can be tacked on to the first one by using the [new_Query_from_query_pair()] function, which also allows you to specify a Boolean operator used to glue the two queries together. As you can see, this process is actually quite simple— the difficulty lies in interpreting user input in order to produce a well-formed query. Luckily, Xapian offers a query parser that can be used for the purpose—we’ll look at it later on and, for the moment, limit ourselves to probabilistic searches using the or operator only. Once all the search terms have been built into a final query, this must be inserted into an enquire context, which can then be used to execute it by calling the enquire_get_mset() function. The latter allows you to choose which rows should be returned, much like a limit clause would on a MySQL query. Query results are returned using an Iterator. This is a very convenient template construct in C++ that becomes… a little less convenient in your PHP scripts. With a proper OOP implementation (particularly in PHP 5, where iterators are part of the Standard PHP Library), this approach would allow us to extract the data using nothing more than a simple foreach loop—but we are stuck with procedural code in PHP 4 and, therefore, we must navigate the result set using a slightly less elegant while loop instead. Let’s take a look at the example in Listing 2. As before, this script essentially follows the process that I just outlined. Note that, once again, we are converting each search term to lowercase before adding to our query—this keeps everything within our application nice and consistent. After executing the query, on line 52

Homo Xapian: A Search for a Better Search...Engine

Listing 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

“string2” => “string3” =>

array “He kept rolling down the stairs”, “Living la vida loca”, “Click the cat has seven lives”);

dl (“xapian.so”); $db = new_database (“db”); $stemmer = new_stem (“english”); if (!isset ($_POST[‘search’]) || $_POST[‘search’] === ‘’) { die (“Please specify a search string.”); } // Create a query parser $query_parser = new_queryparser(); // Set the stemmer and turn on the stemming strategy queryparser_set_stemmer ($query_parser, $stemmer); queryparser_set_stemming_strategy ($query_parser, 1); // Set the database queryparser_set_database ($query_parser, $db); // Parse and create the query $query = queryparser_parse_query ($query_parser, $_POST[‘search’]); echo “Searching for “ . htmlentities ($_POST[‘search’]) . “

”; $enq = new_enquire ($db); enquire_set_query ($enq, $query); $result_set = enquire_get_mset ($enq, 0, 10); echo mset_get_matches_estimated($result_set) . “ results found

”; $item = mset_begin ($result_set); while (! msetiterator_equals ($item, mset_end ($result_set))) { echo htmlentities ($search_strings[document_get_data ( msetiterator_get_document ($item))]) . “
[ID: “ . msetiterator_get_docid ($item) . “ - Data: “ . htmlentities (document_get_data ( msetiterator_get_document ($item))) . “ - Match: “ . msetiterator_get_percent ($item) . “]
”; msetiterator_next($item); } ?>

34 we output the number of estimated matches by calling [mset_get_matches_estimated()] and then enter a [while()] loop, where we cycle through our result set. For each document that our query returned, we extract the unique document ID (which you’ll rarely need since you can store your own arbitrary metadata), followed by the match probability that search engine calculated and, finally, the metadata that we associated with the document itself, which we then use to retrieve our original text string. Figure 1 shows the search engine at work. As you can see, I searched for the string “I am,” which returned two matches as expected. Using a Stemming Algorithm Even though we have a functioning search engine, it is far from being flexible. For example, our current efforts at neutralizing certain semantically insignificant differences in the search terms are very limited: while we are

June 2005

●

PHP Architect

●

www.phparch.com

FEATURE converting every word to lowercase, that doesn’t help us one bit when it comes to performing a search that is based on the root of a word rather than on its declination. For example, a user looking for “debug” will expect the search engine to match words like “debugged,” “debugger” and “debugging”—all requirements of which our application is blissfully unaware. Xapian, however, is a classy tool and, therefore, it comes bundles with a stemmer—a utility capable of applying the appropriate rules to reduce a word to its root (or “stem”) in a given language. The stemmer that comes with Xapian must be invoked separately (so that the semantic rules that apply to text don’t have to interfere with other types of data) and supports a variety of different languages. Its use is incredibly simple: all you need to do is instantiate a new stemmer by calling the new_stem() function, and then call the stem_stem_word() function on every word that needs to be indexed or used as a search term before passing it on to Xapian itself. As you can see in Listings 3 and 4, this changes very little as far as our scripts are concerned. However, when we go to execute a query, the results change quite dramatically. In Figure 2, for example, you can see that searching for the word “roll” returns one entry that actually contains the word “rolling,” thus making our search engine much more powerful. It is important to keep in mind that, like all software that deals with human languages, the stemmer is far from perfect. Consider, for example, the result set shown in Figure 3. As you can see, I was searching for the word “life” and would have reasonably expected a match against the string “Click the cat has nine lives,” since “lives” is the plural of “life.” Unfortunately, this word does not follow the normal rules of the English language and, therefore, the stemmer is incapable of dealing with it in a proper way. On the other hand, searching for “lives” matches both the string above and “Living la vida loca”—proof that, while not perfect, the stemmer does work in most cases. Taking Advantage of the Query Parser As I mentioned earlier, Xapian includes a query parser capable of interpreting a complex set of syntactical rules (shown in Figure 4) and returning a query object ready to be fed to the search engine. You will probably find that the full functionality offered by the parser is actually overkill for all but the most advanced of users, so this particular little utility can come in very handy when you don’t want to write your own parser—an operation that, in PHP, can be as dull as it is complicated. Using the query parser in your script is extremely easy. All you need to do is instantiate it and initialize it with a stemmer (if you need to use one), then feed it the

53

FEATURE

Homo Xapian: A Search for a Better Search...Engine

Figure 4

OPERATOR

DESCRIPTION

AND

Matches documents that are matched by both operands

OR

Matches documents that are matched by either operand

NOT

Matches documents that are matched only by the left operand

XOR

Matches documents that are matched by one of the operands, but not by both

( and )

Allows for the sub-grouping of expressions

+ and -

Unary operators. Match terms that contain all operands prefixed by a plus sign and none of the words prefixed by a minus sign. For example "+Marco -Tabini"

NEAR

Matches documents in which the two operands are within ten words of each other

““

Allows for phrase search

databases at the same time, returning a seamless result set that incorporates documents from all the various schemas. You can even run a query through multiple machines across a network—each machine can return its own results, which will be combined with all the others into a single set. These distributed execution features alone give you nearly unlimited scalability. In terms of performance, Xapian’s specialization—if you’re used to the speed of MySQL’s full-text engine, particularly with a very large data set, you’ll be blown away. On an adequately-equipped machine, Xapian provides all the robustness needed by a large-scale enterprise application with none of the costs normally associated with such software. The one area where you’ll find Xapian a bit frustrating is documentation. While the entire API is fully documented, it is not always easy to figure out exactly how things work, because there are very few examples. Still, I had no problem grasping the basics and, with a bit of research, I managed to get my search engine going in no time at all. Considering how much time I wasted trying to build a search engine with tools that were not nearly as good as they claimed to be, the effort that it takes to get Xapian up and running is minimal—particularly if you consider the results.

search string and obtain a query object in return— that’s it! Let’s take a look at how this is done in practice. Listing 5 shows our little search engine modified so that the query parser has replaced the original parser written in PHP. As you can see, the new_queryparser() function creates a new query parser instance, which is then initialized by providing it with a stemmer (qqueryparser_set_stemmer()) and a stemming strategy (qqueryparser_set_stemming_strategy()). In this case, we are instructing the parser to stem all words unless they end in a dot; this parameter can also be set to zero (meaning that no words are ever stemmed) or to two (in which case all words are always stemmed). Finally, a call to queryparser_set_database() ensures that the query parser is accessing our database, while queryparser_parse_query() interprets the search string and returns a query object that we can handle in the usual manner. Where to Go From Here Despite the fact that even by just implementing the material that I presented in this article you will end up with a rather powerful and very scalable search engine, we barely scratched the surface of what Xapian can do. One of the most interesting aspects of the search engine is the ability to execute a query on multiple

June 2005

●

PHP Architect

●

www.phparch.com

About the Author

?>

Marco is the Publisher of (and a frequent contributor to) php|architect. When not posting bogus bugs to the PHP website just for the heck of it, he can be found trying to hack his computer into submission. You can write to him at [email protected].

To Discuss this article: http://forums.phparch.com/229

54

TEST PATTERN

T E S T

P A T T E R N

The Construction Industry by Marcus Baker

We write slices of applications. The PHP architecture involves writing code on a page by page basis. At the start of every page we have to create every object and at the end of the script they are all torn down. This constant setting up and tearing down makes object construction very important to the PHPer. It’s not always an easy task though, as if one object can create another you have introduced a dependency just as much as if one object uses another. As dependencies can make your code tangled, so what are the options?

W

hen we talk about dependencies, we usually think of one object using another—but there is another equally important type of dependency. Unless we are going to create every object in the top level script, some of our other objects will be creating other objects too. This process is called instantiation. Like any dependency, you want the connection to be as minimal as possible and carefully chosen. In PHP, we do a lot of construction, so it’s worth knowing the options. This is going to be a simple list, working through the most common ways one object can create another. I am only going to list object oriented methods, but in spite of the simplicity, there are a lot of patterns. Not a very exciting topic? Well the times they are ’a changin’, so bear with me. Because I cannot give an exhaustive example for each pattern, I am going to use the same example for each one. Not only that, but I am going to abbreviate it heavily and ask your imagination to fill in the gaps. To help you out though, I will describe the example I am going to use throughout very carefully now. We have a template class, called Template, that can have substitutions added to a specially marked up HTML file. We won’t worry too much about the substitution details except to say that these will mostly be simple name and value pairs. The main part is the Template::paint() method, as this will actually print

June 2005

●

PHP Architect

●

www.phparch.com

the combined content to the browser. It’s not that simple though, because we would like a string such as “RR&B’s legacy to the world” to be sent to the browser as “RR&B's legacy to the world”. Being sticklers for clean code, we want strict XHTML, but we don’t want all of that tedious conversion code cluttering up our nice simple Template class. Because of this, we’ll move all of this XHTML functionality into a class called XhtmlWriter. It will be a simple class with just a single XhtmlWriter::write() method. When you send a string to this method, the string pops onto your browser all neatly converted. The Template::paint() method will use the XhtmlWriter rather than print directly. I hope that you can already visualize how to implement these two classes, they are not complicated after all, and that we can move on to the bigger issue. The big question is: how will the Template get hold of an instance of the writer?

REQUIREMENTS PHP

5

OS

Any

Other Software

None

Code Directory

construction

55

TEST PATTERN

The Construction Industry

1. Just Use “new” The simplest approach is to create the writer at the point of need. In our case this is in the paint() method itself... class Template { ... function paint() { $writer = new XhtmlWriter(); ... } }

This makes using the Template very easy... $template = new Template(); $template->set(‘name’, ‘Marcus’); $template->paint();

Because the creation of the writer is sealed inside the paint() method, as programmers we are completely unaware of it, and so the client code is very clean. This is the tightest possible interaction between the two objects. The writer lives just long enough to do its job, and no longer. Unfortunately, this means we have no control over this process. What if we want a simple plain text writer, or perhaps a Rich Text Format (RTF) writer? More importantly, what happens when we want to test the Template class? Any test will cause the template object to start writing out text, probably messing up the display of our test tools, and it’s not so easy to test output once it has been sent. With this implementation, there is nothing we can do; it’s XHTML or nothing. 2. Parameterization The lightest touch is to have the two classes in contact for the duration of the method call only. We do that by passing one object into another... $template = new Template(); $template->set(‘name’, ‘Marcus’); $template->paint(new XhtmlWriter());

It is now the caller’s responsibility to create the writer. The good news is that we can create any writer that we want. The bad news is that we will have to do it for every call to Template::paint() which will get annoying if there are a lot of them. It will be especially unfortunate if one part of our code creates an XhtmlWriter() and another part creates an RtfWriter by mistake. The burden is pushed to the client code. Despite these minor pitfalls, this is still the simplest way to go. The code is so explicit and emphatic it is very difficult to introduce bugs. I always use this approach unless I have a good reason not to. 3. Passing to the Constructor This is an object oriented programmer’s mainstay... $template = &new Template(new XhtmlWriter()); $template->set(‘name’, ‘Marcus’);

June 2005

●

PHP Architect

●

www.phparch.com

$template->paint();

Here we pass the writer into the Template constructor by reference. This way the Template gets a writer that it keeps, tucked away, until it is needed. This is obviously handy if we have to call Template::paint() more than once, as we only have to specify the writer once for both calls. This mechanism is very common; a number of patterns implement this arrangement: Strategy, Observer, Bridge, Adapter and Proxy for example. We didn’t actually have to use the constructor for this. We could have used a setter instead, but when you do that, you have to make sure that the setter is called before the first use of the object. Being forced to do one thing before another, and having to remember to do it yourself is called temporal coupling. By using the constructor, we make sure that the writer is always created before it is used. Because we pass the object in by reference, we maintain a ghostly connection with it. You could take the Template object over to one side of the room and the XhtmlWriter over to another, metaphorically speaking of course, and Template::paint() would still write messages to the writer. This is the essence of the Observer pattern. The writer is observing the template here, waiting for messages. Another way of looking at it is to think of the writer modifying the behaviour of the template. Right now, it is equivalent to an “XhtmlTemplate”, but if we plug in an RtfWriter we will, in effect, get an “RtfTemplate” object, instead. This is the essence of the Strategy pattern (covered in the both the January and February issues of PHP Architect). The only downside is that the connection is always something that has to be thought about. However, any real world code has to have some connections and, done this way, everything is controlled and very obvious. This technique gets a big thumbs-up. 4. An Internal Factory Method If you are still hankering after the very clean interface of our first example, then this may be for you. Instead of using the new keyword directly, we can use an internal factory method called Template::createWriter(). We still use new, but we’ve created an additional intercept around it... class Template { ... function paint() { $writer = $this->createWriter(); ... } protected function createWriter() { return new XhtmlWriter(); } }

A factory method is really any method that creates

56

The Construction Industry

another object. What’s the point of this? Well, a problem with the hard coded version was that we could not change the writer. Now we can change the writer by subclassing the Template... class RtfTemplate extends Template { function createWriter() { return new RtfWriter(); } }

That was why we marked the method as protected. It was so that we could knock it out when subclassing. Although this adds some measure of flexibility, it is actually a fairly limited technique. Single inheritance means that we can only apply this trick once. If Template already inherits from another class, then things will get tangled pretty quickly. If in doubt, I prefer one of the passing techniques above. One case where the subclassing trick is useful, though, is for making legacy code test-friendly. An essential first step in testing a class is to isolate it from any other class. You don’t want any side effects from your tests and you want complete control of the environment in which the class is running. This leads to a chicken and egg situation. You don’t want to edit the code until it is under tests, but you cannot get it under tests because of all of the strands and dependencies. A way out is to replace all of the construction using new with the little factory methods. This is a trivial change that can be safely done by eye. When it comes time to test the Template, you can now subclass it in such a way that a fake version of a writer is used instead. 5. The Big Guns: AbstractFactory This is a highly factored solution and will solve even the most complicated cases. Creating the factory method internally required subclassing to change it, so instead we’ll pass an object that contains the factory method, directly into our class... $template = new Template(new WebServer()); $template->set(‘name’, ‘Marcus’); $template->paint(); class Template { ... function paint() { $writer = $this->environment>createWriter(); ... } }

$this->environment The variable, in the Template::paint() method is the WebServer object previously passed into the Template constructor. This implements the Strategy approach, as discussed earlier, but adds an extra factory step. For our trivial example, this is obviously an insanely complex solution and complete overkill. To have AbstractFactory make sense we must imagine additional complications.

June 2005

●

PHP Architect

●

www.phparch.com

TEST PATTERN We’ll pretend that there is another processing step in our application and that it takes the $_POST parameters, wrapped up in a Request object. Just like our Template class, the other imaginary code uses our factory class to create all of its objects. Thus, our WebServer class currently looks like this... class WebServer { function createWriter() { ... } function createRequest() { ... } }

To make things easy for ourselves, pretend that we instantiate this object once, at the start of the web page. Now look at this class... class CommandLine { function createWriter() { ... } function createRequest() { ... } }

It creates a TextWriter and gets the request data from the command line arguments. As far as other objects are concerned though, these are details. All they care about is that they get some abstract kind of factory. Think about that for a second. We can have our code change from a web based application into a command line tool by changing one class name. This is the real power of object orientation, but complexity-wise, it comes at a price. 6. Static Factory So far, we have been passing a lot of objects around. Another way to make the factory method available is to have it static... class WebServer { static function createWriter() { return new XhtmlWriter(); } } class Template { ... function paint() { $writer = WebServer::createWriter(); ... } }

Of course the static method could be a global function as well as they are pretty much equivalent. The only advantage this approach gives is that you could change the XhtmlWriter class name without affecting the rest of the code. Well you can do that with a search and replace operation, so this benefit is a mirage. The problem with static methods is that they fix the class name, robbing you of polymorphism. You can create a different class depending on the parameters, but the set of available options is sealed. 7. The Evil Global As if statics were not bad enough, I am going to cover

57

The Construction Industry

something even worse... global $writer; $writer = new XhtmlWriter(); class Template { ... function paint() { global $writer; ... } }

Once you resort to globals, the chances of accidents happening heads rapidly to 100% on any project that is remotely complicated. Besides some other method

TEST PATTERN on destruction. If want all of your queries going out in a single transaction, it makes sense to ensure that a second one cannot be created to spoil the show. Here is a typical PHP5 implementation: class XhtmlWriter { ... function instance() { static $writer; if (! isset($writer)) { $writer = new XhtmlWriter(); } return $writer; } }

“ Unless we are going to create every object in the top level script, some of our other objects will be creating other objects too. ” overwriting the global, you have to make sure it is properly set up, once only, before you use it. The usual reason for using a global is to cut down on the clutter of passing the object around. Is passing the object, or at least a factory for that object, much work? You think so? Have you actually tried it? In a PHP web application, each page request usually touches only a slice of the application. The need to keep objects global should be rare. The real evil of globals, though, is the combination of action at a distance and the availability of the object to every other part of the code. Once the value has been changed, you have no way of knowing which method in which class did the deed. Globals also create havoc with testing, as one run will interfere with the next. To prevent accidents, programmers will come up with all sorts of naming conventions to alert the danger, not realizing that they are creating more work than if they just added some intelligent design. When the disaster occurs anyway, debugging is a nightmare. 8. The Singleton This is a fiendishly clever pattern to get around the limitations of a global variable, and yet still have a single instance that is accessible to all. Basically, it is a static method that only returns one instance that is, itself, static. There are all sorts of reasons why you might want to guarantee something is only created once and then reused. A database connection is an example when used in a simple system. The connection has to open a transaction on construction and close it again

June 2005

●

PHP Architect

●

www.phparch.com

When we make a call to XhtmlWriter::instance(), we receive a freshly created XhtmlWriter instance, as a result of the method block execution. Because the $writer variable is static, it survives the first invocations and is still around when we ask for it again. This trick of keeping a variable alive inside a code block is called a “closure”. We keep passing back references to this single variable, a variable that cannot be directly accessed except by the code block that created it. By getting the only instance this way, we ensure that it cannot be overwritten and that it is guaranteed to be instantiated correctly... class Template { function paint() { $writer = XhtmlWriter::instance(); ... } }

A clever little pattern, but the static method makes this pattern very inflexible. You have to rewrite the code to change the type of instance. It also makes the application difficult to test, because that single instance is impossible to reset between each test. Code that you cannot test is certainly broken, and the Singleton, these days, has a bad reputation. 9. The Registry This is a major upgrade to the Singleton, but is much more complicated to code. Essentially the Registry is a singleton itself, but acts as a glorified hash into which you can stash other global resources. For example, we

58

TEST PATTERN

The Construction Industry

could start our application with: $registry = Registry::instance(); $registry->setWriter(new XhtmlWriter());

This is similar to just making the writer a global, right? Well sort of, but when we go through a method call, we can add additional protection. For example, we can have the setter throw an exception if the object is accidentally set up twice. In our template we pick up the XhtmlWriter instance the same way: class Template { ... function paint() { $registry = Registry::instance(); $writer = $registry->getWriter(); ... } }

The Registry is not a particularly clever pattern, but it is an effective one. By replacing the global keyword with your own class, you can create your own system just for your application. For example, we could set a writer factory of some kind, rather than just a single writer, if that is what we wanted. Alternately, we could add a method to allow a test version of the writer to be used when testing. The power of the Registry pattern comes from its customization, but we are still messing with essentially global information. If you must have global data, then at least do it with a Registry. If you catch yourself using the Registry in more than a few places though, you should pinch yourself and try to pass some of this information around. Your code will be easier to understand if you do. 10. Dependency Injection A dependency injector is basically a replacement for the new operator. You give it a class name, and it returns an instance of that class. I have had to write a demonstration version of the tool here, because most of the DI tools are for Java right now (e.g. “Pico”, “Spring” or “Needle” for Ruby). Some PHP versions are in the works, but are unfinished at the time of writing. My very minimal framework, nicknamed “Phemto”, exists only as Listing 1. Here it is in action: $injector = new Injector(); $injector->register(‘XhtmlWriter’);

This sets up the dependency injector to deliver our writer: class Template { ... function paint() { $injector = new Injector(); $writer = $injector->create(‘XhtmlWriter’); } }

If this was all there was to it, then this would be a com-

June 2005

●

PHP Architect

●

www.phparch.com

plete waste of time. Luckily, there are two wrinkles that make this a very powerful technique. The first is that the request for an instance can be very flexible. Suppose our XhtmlWriter had been written like this: class XhtmlWriter implements Writer { ... }

The injector is a heavy user of reflection and will know that by registering the XhtmlWriter, we actually have a class that can fulfill the role of a Writer interface. This means that we can write our Template::paint() method like so: class Template { ... function paint() { $injector = new Injector(); $writer = $injector->create(‘Writer’); ... } }

In other words, we can wire up our program using only Listing 1 1 registry[$class] = $class; 7 $reflection = new ReflectionClass($class); 8 foreach ($reflection->getInterfaces() as $interface) { 9 $this->registry[$interface->getName()] = $class; 10 } 11 } 12 13 function create($interface) { 14 if (! isset($this->registry[$interface])) { 15 throw new Exception(“No class registered for interface $interface”); 16 } 17 $class = $this->registry[$interface]; 18 $dependencies = $this>findConstructorDependencies($class); 19 return $this->instantiate($class, $dependencies); 20 } 21 22 private function instantiate($class, $dependencies) { 23 $code = “”; 24 for ($i = 0; $i < count($dependencies); $i++) { 25 $objects[$i] = $this->create($dependencies[$i]); 26 if ($i > 0) { 27 $code .= “, “; 28 } 29 $code .= “\$objects[$i]”; 30 } 31 $code = “\$object = new $class($code);”; 32 eval($code); 33 return $object; 34 } 35 36 private function findConstructorDependencies($class) { 37 try { 38 $reflection = new ReflectionClass($class); 39 if ($constructor = $reflection->getConstructor()) { 40 $parameters = $constructor->getParameters(); 41 for ($i = 0; $i < count($parameters); $i++) { 42 $interfaces[$i] = $parameters[$i]>getClass()->getName(); 43 } 44 return $interfaces; 45 } 46 } catch (Exception $exception) { 47 } 48 return array(); 49 } 50 } 51 ?>

59

TEST PATTERN

The Construction Industry

interfaces. Replacing our XhtmlWriter with another can be done at run time without any obvious change of code, which is handy for testing, if nothing else. The Template part of the code doesn’t even need to know about the XhtmlWriter class, even though it was responsible for instantiating it. The Template knows about the injector, but we can tackle that next. In case the power of this is still dipping below your radar, here is another trick. Remember our third example where we passed a new writer in the constructor of Template. First, a reflection-friendly Template class definition: Class Template { function __construct(Writer $writer) { ... } ... }

Here is “constructor based dependency injection” in action:

also automatically created an XhtmlWriter instance to fulfill the requirement of the constructor. In other words, we have been able to instantiate a Template without any knowledge of its dependencies, at all, yet, we still retain full control of all of its components, and this is all figured out by the injector. Because the dependencies were dealt with on construction, the injector is now invisible to the Template code as well. This is serious decoupling, and has the power to manage large applications and frameworks. To describe dependency injection as a hot topic right now, would be something of an understatement.

About the Author

?>

Marcus Baker works at Wordtracker (www.wordtracker.com) as Head of Technical, where his responsibilities include the development of applications for mining Internet search engine data. His previous work includes telephony and robotics. Marcus is the lead developer of the SimpleTest project, which is available on Sourceforge. He's also a big fan of eXtreme programming, which he has been practising for about two years.

$injector = new Injector(); $injector->register(‘Template’); $injector->register(‘XhtmlWriter’); $template = $injector->create(‘Template’);

That last line didn’t just create a Template instance, it

June 2005

●

PHP Architect

●

www.phparch.com

To Discuss this article:

http://forums.phparch.com/209

60

PRODUCT REVIEW

P R O D U C T

R E V I E W

Agata 7 Open-Source Search Technology by Peter B. MacIntyre

Figure 1

T

his month I am reviewing the product called Agata Report Generator. It is an open source product with a home in Brazil. If you look at the credits page for this product, you will see this list of the supporting cast of tools and products used to build it: • PHP (www.php.net); • PHP-GTK (gtk.php.net); • PEAR (pear.php.net); • PhpDocWriter ( http://phpdocwriter. sourceforge.net); • FPDF (www.fpdf.org); • BarCode128 (GuinuX); • JPGraph (www.aditus.nu/jpgraph); • Ximian Icons (http://www.ximian.com);

June 2005

●

PHP Architect

●

www.phparch.com

PRODUCT INFORMATION PHP

4+

OS

Any

Product Version

7

Price

FREE!

Web Address

http://www.agata.org.br

61

Agata 7: Open-Source Search Technology

ago now when I was developing a rather complex PHP system, and at the time it seemed to be quite immature. Now, I am looking at it again and it seems to have grown up quite nicely. Figure 1 shows the start up screen after the product has been installed. Notice that this is a command line (DOS in Windows) screen that starts up the product. This means that the product is done with the PHP-GTK module (among the list shown above).

Figure 2

Figure 3

So as you can see this is a rather unique product that has been built with quite a few other open source tools. So, what does their web site have to say in summary of what this product can do? Agata Report is a cross-platform database reporting tool with graph generation and a query tool that allows you to get data from PostgreSQL, MySQL, SQLite, Oracle, DB2, MS-SQL, Informix,

June 2005

●

PRODUCT REVIEW

PHP Architect

●

InterBase, Sybase, or Frontbase and export that data as PostScript, plain text, HTML, XML, PDF, or spreadsheet (CSV) formats through its graphical interface. You can also define levels, subtotals, and a grand total for the report, merge the data into a document, generate address labels, or even generate a complete ER-diagram from your database. Wow, that is a mouth-full. I looked at this product a few years

www.phparch.com

The 50 Cent Tour After the user responds to the opening screen with a language selection and a possible theme, the interface window appears (Figure 2). This is a relatively clean interface and not too difficult to figure out on your own. I say figure out on your own because the most glaring point against this product is the complete lack of any documentation. In my exchange of a few emails with the product "owner" (Pablo), I was informed that the major effort into the product is not the documentation. He acknowledges the fact that there is little to no documentation and is focusing on stabilizing version 7 for the time being. If you go on the Agata web site you will see an invitation for someone to do the documentation, so if any readers of this article have some spare time, they could contribute to a good open source project in this manner. Now, let's get back to the main user interface screen. The first thing you will have to do is connect to a database. Since this is a reporting engine, it needs a source of data on which to report. Using the File -> Connect to Database menu option you will be taken to the screen shown in Figure 3. Here, you make your connection by providing the requested connection information and you are returned to the main

62

PRODUCT REVIEW

Agata 7: Open-Source Search Technology

face is really doing is building a SQL Select statement for use in the report. Figure 5 shows that the grouping bands and sub-totaling is done separately. This is different than the SQL GROUP BY clause, so be careful of that distinction. I liked this interface, although it did take me a little time to play with and figure out because the methods used to navigate through the features were not consistent; some times you double-click, sometimes you use a drop-down box, and sometimes you select a field and click a button. Figure 6 shows the preview of my defined report.

Figure 4

Figure 5

screen. Once this connection has been established, the main screen is populated with table names from the database to which you are connected. Now that the connection has been established, it's time to select information on which to build a report. I chose my small database

June 2005

●

PHP Architect

●

that contains a practice schedule for the Junior High School basketball team that I coached. Figures 4 and 5 show some of the report definition already built. I added page headers and footers (these don't show up in the report preview screen), and group and page totals. As you can see, what this inter-

www.phparch.com

Summary This product has other features that I am not covering in this review. There are many ways to preview and save your designed reportsnamely XML, OpenOffice, PDF, CSV, TEXT, and HTML. There is even a way to tie these report definitions into a PHP application. So, be sure to test these features out when you are looking into the product for yourself. Other features that were noticed were graphical reports (done with the help of the JPGraph PHP library), and both label and bar code generation features. This product is always getting closer to becoming a prime time challenger to Crystal Reports, but it is not quite there yet. Now, don't get me wrong; this application is very useful and versatile and is a great testament to the integration of PHP extension tools being used in concert. It certainly has come a long way in maturity since I first laid eyes on it a few years ago. Kudos goes out to Pablo-who is the main developer of this product-for conceiving it, and maintaining it in all its complexities. The feature set appears to be lending itself to a robust set of tools; perhaps the

63

Agata 7: Open-Source Search Technology

PRODUCT REVIEW developers have bitten off a little more than they can chew? My main problem with this product is, as I mentioned earlier, is that it has a great need for some quality documentation. At the very least, it should have a short tutorial on how to connect to a database and how to define some of the more basic report types so that the prospective user does not have to give up 3 days in trail and error.

Figure 6

I give this product 3 out of 5 stars.

About the Author

?>

Peter MacIntyre lives and works in Prince Edward Island, Canada. He has been and editor with php|architect since September 2003. Peter’s web site is at http://paladin-bs.com

June 2005

●

PHP Architect

●

www.phparch.com

64

You’ll never know what we’ll come up with next For existing subscribers

NEW

Upgrade to the Print edition and save!

LOWER PRICE! Login to your account for more details.

php|architect

Visit: http://www.phparch.com/print for more information or to subscribe online.

The Magazine For PHP Professionals

Address: _________________________________________ City: _____________________________________________ State/Province: ____________________________________

E!

RIC

RP

NE

W

*US Pricing is approximate and for illustration purposes only.

WE

Name: ____________________________________________

Your charge will appear under the name "Marco Tabini & Associates, Inc." Please allow up to 4 to 6 weeks for your subscription to be established and your first issue to be mailed to you.

Choose a Subscription type:

Canada/USA International Air Combo edition add-on (print + PDF edition)

LO

php|architect Subscription Dept. P.O. Box 54526 1771 Avenue Road Toronto, ON M5M 4N5 Canada

$ 77.99 CAD $105.19 CAD $ 14.00 CAD

($59.99 US*) ($80.89 US*) ($10.00 US)

ZIP/Postal Code: ___________________________________ Country: ___________________________________________ Payment type: VISA Mastercard

American Express

Credit Card Number:________________________________ Expiration Date: _____________________________________

Signature:

Date:

*By signing this order form, you agree that we will charge your account in Canadian dollars for the “CAD” amounts indicated above. Because of fluctuations in the exchange rates, the actual amount charged in your currency on your credit card statement may vary slightly.

E-mail address: ______________________________________ Phone Number: ____________________________________

To subscribe via snail mail - please detach/copy this form, fill it out and mail to the address above or fax to +1-416-630-5057

EXIT(0);

Tales from the Script e x i t ( 0 ) ;

by Marco Tabini

A

fter much consideration— most of it in my own bed, where I finally slept again for the first time in two weeks last night—I have come to the conclusion that scientists at Miami International Airport must have run out of mice. The rodent shortcoming, in turn, must have forced them to shift their experiments on the third most intelligent life form on the planet Earth: man (the other one being, as all fans of Douglas Adams known, dolphins). This has to be the only reasonable explanation for the absolute maze that I had to walk through to get from gate A1 to A26 and catch my connection flight on the way back to Toronto from Cancun. Not that there is anything reasonable about flight travel anyway—these days, airport procedures seem to be designed by some bureaucrat who, having drank way too much one night, just happened to fall on his keyboard in a particularly clever way and type something that, while linguistically accurate, makes no sense whatsoever to anyone but another bureaucrat. Don’t let the apparent closeness (of the asort() kind) of A1 to A26

June 2005

●

PHP Architect

●

fool you, either—the two gates are a good thirty minutes’ walk away from each other and, most likely, reside in two different area codes. In fact, they probably reside in two different countries, which explains why, fresh from an international flight where a customs officer has rummaged through your dirty clothes of ten days and a half-hour pleasant stroll through the hermetically-sealed “connecting flight corridor,” you have to go through security all over again. Take off your shoes, pull out the laptop from the bag, show that your camera is not really a thinly-disguised atomic device and you’re well on your way to another three hours of over-regulated cattle-class flying. And in case you’re traveling with your brother and you’re both carrying an LCD projector of the same brand and model, you must really be twins—no matter if your passports say you were born three years apart (yes, this really happened; I kid you not). Honestly, if you happen to read this and live in Florida, please write your congressman—the federal government should, at the very least, share the results of the experiment with the rest of us.

www.phparch.com

Meanwhile, back in Mexico, science and technology came together in a large jug of industrial cleaner that I found lying around while I was doing my rounds at php|tropics. According to the label (in Spanish, of course), that particular chemical can be used to either clean stainless steel surfaces or to disinfect fruits and vegetables (but not for cleaning porous materials like porcelain, which could get badly corroded). That bit of scientific breakthrough, however, is nothing compared to the fact that the Airport Authority of Cancun has answered the question “how can we afford a $500,000 automated parking system?” by hiring some guy who simply stands there and pulls up a chain across the road until you’ve shown him your validated parking ticket. Unless you want your car ripped in half like a slightly oversized can of sardines, this method is just as effective and likely much cheaper than your typical American (or Canadian) solution. I love traveling—it’s what makes life worthwhile.

php|a

67

Can’t stop thinking about PHP? Write for us! Visit us at http://www.phparch.com/writeforus.php