The example most programmers will understand today is with a yacc-like tool (or lex, for that matter):

y.tab.c y.tab.h : whatever.yacc

       $(YACC) $(YFLAGS) $<

If you’ve ever used yacc, or the GNU equivalent, bison, or any other code generator, you’ll understand the pattern. There’s a set of input files, usually either just one, or one plus some other files generated by a lexer generator. And the input files are read by the code generator tool, and the code generator tool produces a bunch of different output files all at the same time.

The syntax that this ‘make’ tool I was reading about used to indicate this was a plus sign:

y.tab.c + y.tab.h : whatever.yacc

And that plus sign was enough to let the ‘make’ tool know what was going on with the simultaneous generation of both files.

The company in question was not a client. It was my second serious job after college. I started working there in the (late) 1980’s. And the ‘make’ tool in question was make, as provided with SunOS 3 and 4. (Not Solaris. That came later.)

Yeah, so?

I told that story for a reason: to point out that this somewhat niche-market problem has been solved, in a widely-available system, using a trivial syntax enhancement. The solution has been around for more than 20 years. And GNU make still gets it wrong.

The latest GNU make version provides a feature that almost solves the same problem. Except that it doesn’t. And that “feature” is a hack.

It’s documented in the manual, but not mentioned in a header in the contents, so you have to just know, or have someone give you a link. Google won’t help you find it. (It’s here.)

And of course the syntax used for the GNU make not-quite-but-maybe-almost hack is: two targets separated by spaces.

That’s right, the GNU version looks like:

a b : input

Which you might find confusing, because the GNU make syntax for two targets that do not all get regenerated at the same time, but instead require the recipe to be executed once for each output looks like:

a b : input

See the difference? Of course not, because at this level of abstraction there is no difference. The rules look exactly the same.

If you don’t know the secret – which is that the run-once or run-multiple-times behavior depends on whether the rule is a pattern rule or not – then you’re SOL, because there’s no visible indicator.

It’s worth pointing out that there’s a method to the madness of the GNU make weenies. The justification, if I were dumb enough to ask for one, would be something like this: the only time this kind of thing ever arises (in C) is when you’re using yacc or rpcgen. (Rpcgen is the reason the Sun guys added the rule, I’m fairly sure.) And yacc/rpcgen always get used in such a way that the filenames always have a common root. So making this incredibly lame hack makes total sense, because nobody ever does this, but if they did they’d be happy it was this way.

Bah, humbug!

I don’t do a lot of C programming any more. There are other languages that are generally much more suited to what I’m trying to do. Some of those languages are “static” languages, like C++, Java, and D. And some of those languages are “dynamic” languages, like sh, perl, and scheme.

But in both camps, you’ll find languages that use directory names and file names to reflect the classes, modules, packages, units, or whatever-it’s-called defined inside them.

This is important because if I’m going to generate code, I want to be free to generate code using file and directory names that are meaningful to, and constrained in, the context of the programming language I’m actually using.

I really, really, really don’t want to have to conform to a naming convention inspired by a tool I’m not using in a language I don’t care about that is the basis for a syntax hack in make!

Ultimately, I think this is an example of a couple of things. First, that the GNU make guys are suffering a vision problem because they’re focused on building a core set of applications, using a small set of languages.

Second, I think they’re afraid to admit that other people, when faced with a similar set of problems as them, have come up with useful solutions. Back when GNU make got started, SunOS was the dominant Unix. And the guys at Sun were doing a fine job of carrying forward the tool suite. But heavens! For GNU to copy a useful feature of a popular tool would be to admit that we aren’t the sole source of clever ideas. Oh, noes!

Thus, we have the open source curse: Gnot Invented Here syndrome.

And also thus – more of a pain in my tuchus, thus – we have a version of GNU make that forces me to jump through a bunch of hoops to accomplish something that I could have done in two lines back in 1988.

Today I was chatting with some folks on IRC, and the subject of binary searching came up. Now, I don’t know if this is going to surprise you, but in the last few years there has been some “movement” in the performance arena as far as thing we just “know”. It turns out, for example, that the “qsort” function isn’t the fastest horse in the race any more. And it turns out that bsearch is getting a little long in the tooth, too.

“Mon dieu! How can this be?,” you ask. Well, it’s templates.

The C++ guys apparently listened to Philip Greenspun, and decided that the way to better performance was to build a lisp interpreter into the compiler. That lisp interpreter is called “template metaprogramming” (TMP), and the languages like C++ that support TMP can use it in some surprising ways.

The key thing to realize here is that there’s a trade-off involved. The old C standard library functions are – no question – about as fast as they can be. But the TMP guys came to the realization that the library functions were just that: functions. And so calling qsort or bsearch means passing in a callback function that takes a couple of pointers and compares whatever they point to.

This is very generic, but it comes at a fairly high cost in terms of run-time. Calling a function to make a comparison is a huge performance lose, especially on a fast CPU with a good sized cache. All that stuff you’ve heard about cache misses and branch prediction and what-not? Well, it becomes relevant if you have to call through a pointer to the compare function, and it gets doubly relevant if your compare function turns around and calls some other function – like strcmp.

So the template version of all this uses the same logic as the hoary old C standard library chestnuts, but it uses templates to dynamically build a special-purpose function for doing just exactly the Quicksort or binary search that you need to do right now.

The obvious down side? Each place where you invoke qsort or bsearch generates a different function. Big code bloat here, folks.

The up side? Breaking through the “call a function to compare two items” barrier. For small types – like int, float, etc. – and for “fully equipped” types like strings, there will be enough code laying around that the compiler can in-line some, if not most or all, of the comparison logic. Comparing strings will always require looping through arrays. But comparing two structs? That’s easy to inline. Comparing two ints? Please!

Crossing Over

So with this in mind, I set out to find the “cross over” point. Put simply, where do the performance curves of linear seach (lsearch) and binary search (bsearch) cross? That is, for N = how many items does it make more sense to use a binary search than to iterate?

Now, here’s what we know about bsearch: it has performance characteristic O( log n ). And we know that lsearch has performance characteristic O( n ). But what does that mean, exactly?

First, every routine has some overhead. When a subroutine is called, you push some information on the stack, initialize some variables, and then do your work. And later, you takes stuff off the stack, free your resources, and return. That’s overhead.

Next, in the process of doing the actual work, you have to perform interim computations. For example, linear search basically requires some mechanism for iterating through the array you are searching. And that’s going to be a pointer or an integer or something. Updating that iterator is part of the structural cost of the algorithm. A bsearch algorithm typically involves computing (low + (high-low)/2). That’s the middle point for the search, and that’s a part of the structural cost.

Those structural costs can be high, sometimes. So saying that an O( log n ) function is faster than an O( n ) function automatically implies a caveat, so long as n is big enough that the structural and overhead costs get washed out. And that’s why I was curious about the cross-over number. Because with templates, the call-a-function-through-a-pointer structural cost is gone. And so my understanding of where the lines cross is no longer valid.

Great! Another ‘D’ in programming…

The language in question isn’t C++, though. It’s D. Now, D is a lousy name for a programming language, just like C is. Because Google is not your friend if you’re searching for “D”. Nor is “file.d” going to be a win, either, because that’s the default name for dependency files in a lot of build systems. So D wasn’t the best possible choice of names, and it can be hard to find info about it. To save you some grief, have a look at http://www.digitalmars.com/d – that’s the “official” D site, if there is such a thing. (For myself, I’ve taken to using “PL/D” as an abbreviation.)

PL/D version 2.0 supports templates. So here’s bsearch, as a can-find type function:

bool bsearch( T, alias cmp = "a < b" )(T[ ] array, T key )
{
	int left = 0;
	int right = array.length;

	while( left + 1 < right ) {

		int middle = ( left + right ) / 2;
		T am = array[ middle ];

		if( binaryFun!( cmp )( key, am ) ) {

			right = middle;
		}
		else {

			left = middle;
		}
	}

	// return a[0] == key
	return ! binaryFun!( cmp )( key, array[ 0 ] )
		&& ! BinaryFun!( cmp )( array[ 0 ], key );
}

Now I know that this version doesn't do early exit. But it's a little more readable to folks who might be new to the whole template thing, especially with PL/D. One thing I like more about D than C++ is that templates are instantiated like foo!( T )( a, b ) instead of foo< T >( a, b ). If nothing else, it's more html friendly.
I won't explain all the neat-o features of the language here - see the link above for that. I will point out that this version generates a binary function using "a < b" by default, and generates a direct call to that function in the comparisons. As a result of the direct (instead of indirect - through a pointer) function calls, the compiler can inline them, resulting in the comparison being made right in the function itself.
The testbed was an array of randomly-generated integers. The driver code looked like this:

	// ========================
	Thread.sleep( 1 );
	timer.start();

	foreach (iter; 0 .. ITERATIONS )
		bsearch(a[], cast(int)(rand() % ELEMENTS));

	timer.stop();
	writefln( "Bsearch: %f usec (avg)", cast( float ) timer.microseconds / ITERATIONS );
	writefln( " .... Cycle: %f usec (avg)", ( timer.microseconds - overhead ) / ITERATIONS );

	// ========================
	Thread.sleep( 1 );
	timer.start();

	foreach (iter; 0 .. ITERATIONS ) {
		int key = rand() % ELEMENTS;
		csearch( &key, &a[ 0 ], ELEMENTS, int.sizeof, &compare_ints );
	}

	timer.stop();
	writefln( "Csearch: %f usec (avg)", cast( float ) timer.microseconds / ITERATIONS );
	writefln( " .... Cycle: %f usec (avg)", ( timer.microseconds - overhead ) / ITERATIONS );

Where most things should be obvious, but I’ll point out that the Thread.sleep calls were for 100 nanoseconds each – just enough to give control back to Windows so it hopefully wouldn’t interrupt the loops. The “csearch” call in the second block is actually a call to the bsearch function in the C standard library – I created an alias for it since I already had a bsearch function.

Overall, for ELEMENTS = 1000, the cycle time for the template based bsearch was about 0.12 usec. The cycle time for the call-a-function C library version was about 0.165 usec. So for trivial comparisons, the template-based version gives a 25+% speed benefit – that’s why I was interested in running the tests!

Obviously, this number depends on the number of comparisons. For ELEMENTS = 100, you’re looking at log n = 7 instead of 10 comparisons – csearch should catch up a little bit. For ELEMENTS = 100,000, it would be 17 instead of 10 and presumably the template version would show even better performance.

The Edge of Seventeen

But that’s not why I started this. I wanted to know how well the template bsearch code would do against lsearch – linearly scanning the array. It seemed like a safe bet that 1,000 elements would be a win for the binary search…and it was. (whew!) But would the same be true for 100 elements? 10?

Well, for me, the number is 17. If ELEMENTS = 17, then lsearching the array (using a template based lsearch) is faster than bsearching the array (using a template based bsearch). (On the other hand, template based lsearch crosses over C-style bsearch at 61 items!)

Those performance numbers depend on a lot of things. Most importantly, they depend on the relative speed of your comparison code versus the structural code of the search function.

What does that mean? Well, take a look at the a program called git-bisect. This program is a shell script that uses a binary search to find a build failure. Bisect knows the list of changes that were made to your source code, so it can pull out a version, build it, test it, and decide whether the problem is in that version or elsewhere.

The issue, though, is that the “comparison function” involves pulling out your 30 million lines of code, running a build, and then running the test that identifies whether the bug is in this version or not. The build and test process takes so much longer than the “add two numbers and divide by two” of the binary search that it’s a no brainer – you’re almost always going to use the bsearch version.

But when the compare is a single CPU instruction, it’s a different story. In that case, the overhead of the bsearch pushes the crossover point far out past where it is in theory. If your compiler is smart enough, and when you’ve got enough memory to allow it to inline these kinds of functions…well, we have to draw a line somewhere.

But I’m doing unit testing, and writing unit tests. In this case, I’m using the boost C++ libraries. That means my tests don’t look like classes, the way they might look if I was using CppUnit or CxxUnit. Instead, they look like macros:

BOOST_AUTO_TEST_CASE( null_ctor ) {

Arena & sut = Simple::Arena();
BOOST_CHECK( sut.capacity() != 0 );
}


The macro expansion is really quite clever, in a Lovecraftian “there are secrets man was not meant to know” fashion. And reading them has certainly improved my knowledge of C++. (But it also permanently lowered my SAN by a few points, I think.)

Anyway, Doxygen offers this command called \test that seems tailor-made for testing.

Well, it’s not. The command description is very “vague,” in that you use it to add entries (paragraphs) to the “test list.”

Some experimentation will show that the apparent use is to document something (a class, a function) and add a bunch of these \test entries, just like adding \param commands. And the entity you are documenting will then have a Test: section in its documentation.

This is totally not what I want. And I’m pretty sure it’s not what anybody doing TDD wants. What I want is to somehow tie my tests, which are in a separate location (different file, different class, different namespace, etc.), back to the class or function that I’m testing.

The best way I’ve found so far to accomplish this is to lie to Doxygen about what is being documented. Here’s my current scheme:

#define BOOST_TEST_DYN_LINK
#include <boost/test/unit_test.hpp>

Arena & sut = Simple::Arena();
BOOST_CHECK( sut.capacity() != 0 );
}


Adding the \class command tells Doxygen to re-open the documentation of the target class. The \test command is then associated with the “right” class, and I include the testcase name in bold as part of the test description.

This still doesn’t generate a link to the right source file (containing the test), and it doesn’t really know anything about the test case. So it’s not a perfect solution. But it does enable me to put my testcase documentation with my testcases, while having the documentation actually show up attached to the right class.

So here’s which.cmd – a script that tries to DWIW.

Note that this script is wrong in one key detail: it searches inside-out. That is, any “foo.COM” will take precedence over any “foo.EXE” even if the .exe version occurs earlier in the PATH.

@echo off
REM Copyright (c) 2010, Austin Hastings.
REM This file may be used for any purpose without restriction.

REM NOTE: This script does not handle multiple entries with different
REM extensions correctly. It returns the first matching EXTENSION, rather
REM than returning the first DIRECTORY with any EXTENSION.

REM Analogous to unix 'which' command, look for a matching runnable
REM in %PATH% and print the location.

SETLOCAL ENABLEDELAYEDEXPANSION

set check_extensions= ;%PATHEXT%

:ext_loop
if "%check_extensions%" == "" goto done

for /F "tokens=1* delims=;" %%E in ( "%check_extensions%" ) do (
	set check_extensions=%%F
	set target=%1%%E
	for %%W in (  !target! ) do set answer=%%~f$PATH:W
)

if "%answer%" == "" goto ext_loop

:done
if "%answer%" == "" goto not_found

@echo %answer%
goto exit

:not_found
@echo No runnable matching '%1' was found.

:exit

My client was reorganizing a development team into an agile mode in order to deal with an outsourcing disaster. It was one of those horror stories that starts out like “We were 3 years in to a 10 month project, when …” I was brought on to help ramp up their build and deployment processes, to get continuous integration and testing systems up, and to help mechanize their deployments.

The impetus for the MySQL unit test framework came from a short term contractor. The one “constant” in the otherwise highly-variable environment had been that the team wasn’t changing. Suddenly, that stopped being true. We had a handful of contractors in for a short term surge, and none of them were familiar with the application, or with SQL (it seemed). The offender was using the CI server as his syntax checker for database scripts. Yikes!

Luckily, the garbage he delivered was syntactically invalid. The mysql command-line client rejected the script and exited with an error status, which caused the CI script to log a failure, and … voila! Everyone credited me with doing a great job of implementing CI, because it was even checking the database stuff. Like a good consultant, I said “Yes. Of course.” Then I ran back to my desk, thinking “Oh, no! I’ve got to come up with a unit test framework for the MySQL stuff, pronto!” And I did.

Solutions

The software was a pretty complex web app, and the target environment was a mixture of Linux and Windows. As a result, when I looked at existing SQL unit test frameworks (dbUnit and sqlUnit, respectively) I had to reject them. They are decent products – and free – but the fact is that they expect certain things about their runtime environment that we couldn’t deliver. We had to build our own MySQL unit test framework, ideally in “pure” MySQL.

Fortunately for me, a nice guy named Giuseppe Maxia maintains a web site at datacharmer.org, and he has written a library of general-purpose MySQL code. (Called the General Purpose Stored Routine library, or gp_sr_lib.) Among the all this well-written, free, already debugged code was a set of testing routines. Thirty minutes into my development effort, I was already 80% done. Woo-hoo!

The GP library test routines are a good start on a mysqlUnit library, and I strongly recommend that if you need such a library you stop reading here and go download the code. Right now.

Rolling (y)our Own

There are a couple of gotchas that you’ll want to watch out for in your own unit testing library. First, the tests should be able to integrate into your CI engine. In particular, you need to decide on and implement some kind of failure mode – either fail when the first problem occurs, or fail after running all test cases. If you’re running the mysql client via some kind of exec call, be aware that while syntax errors will cause the client to exit with a failure code, other errors will not cause a failure. Instead, the client prints a diagnostic and returns zero. What’s more, the syntax does not support a deliberate abort from within the SQL engine.

In my own case, I chose to follow the model that Maxia has provided, and NOT fail as soon as a test fails. Rather, my code collected the results of all the tests that were run, and summarized them. In order to get “fail on error” behavior out of the mysql command line client, I coded a routine that would perform a “SELECT force_mysql_client_abort FROM no_such_table”. This definitely does cause the client to abort, at least until some developer creates a table called “no_such_table.” Then I wrapped the test cases in an “ignore failure” script that called the summarizer after all the cases had run, which in turn would call the abort routine if the error count was non-zero.

Another thing I chose to “fix” was the routine names. Maxia’s code includes both functions and procedures, but the names are not really in keeping with the style used in the various xUnit test frameworks. The library contains a procedure for asserting the existence of a database table. It is called as:

CALL check_table('db name', 'table name');


I opted to “embrace and extend” the library. I wrapped these types of checks into functions, like this:


set_database('db name');
assert_table_exists('table name');
assert_table_exists_in_db('other table', 'other db');


And in general, wherever there is a straight-ahead assert, I also added an “assert_not”. In terms of effect, there is no real difference in behavior between Maxia’s naming and my own. But I felt that the developers, who were using jUnit and nUnit, would be more comfortable learning a set of routines that had a similar structure and style.

Going Most of the Way: DDL

I tried to make a list of the various things that SQL scripts might do. The most obvious — and this probably accounted for two thirds or more of the scripts we had — is DDL statements. DDL (Data Definition Language) is that subset of SQL that relates to defining databases, tables, indexes, triggers, and the like. While the most frequently executed SQL statements are DML (Data Manipulation Language), DDL is where it’s at for most of the human-generated script files: CREATE TABLE, ADD INDEX, etc.

The great thing is that the assertions for these statements are trivial to write. Maxia’s library includes the CHECK_TABLE procedure, and some code for looking up procedures and functions. Adding other stuff, like keys and constraints, is pretty straightforward. By the end of the day you should have a long list of functions like assert_table_exists, assert_index_exists, assert_database_exists, and assert_column_has_type.

Deploy Early

At this point, you’re ready to go live. Except that you need to build a framework to run these tests. That’s going to depend a lot on your deployment process, and on your particular CI environment. Sorry. The simplest idea would be to write a script that matches the file names of test files, and runs them one at a time. Call that script from CI, and make sure the exit status is meaningful.

Forget the next couple of sections, because most of the database changes you’re likely to see are DDL. Developers and DBAs do a lot of adding fields, adding indexes, and sometimes adding tables. If you have just the existence assertions for each kind of object, your library will cover two-thirds or more of the statements being delivered. That’s not a trivial amount of test coverage for a day or two of work. Ship it! Your developers will need some training, and you’ll need to beat on the thing once it’s deployed to your CI servers to make sure that the errors are coming out correctly.

Sell to the Development Team

I recommend that you write some simple tests of things you know to be true. Confirm the positive cases first. Then sit down with a smart, involved developer and pair develop some unit tests for code he hasn’t written yet. Do it the way you want the developers to do it, including deciding how to handle assertions that don’t exist yet.

In our case, the DBAs had already established a naming convention for database update files. I chose to duplicate the file names, which would look like ####_description.sql, changing the .sql extension to .test.sql. This let the testing framework associate the test with the delivered change. When running a “from zero” deployment, the test framework would then run TEST-CHANGE-TEST and expect the first test to fail and the second to pass. This demonstrated that the test was meaningful for the change (remember, the change is supposed to repair a shortcoming that is demonstrated by the test).

Once you’ve got buy-in from your chosen developer, sit down and develop a presentation that the two of you can give. Point to specific examples of blown deployments as reasons for unit testing the SQL. Then get the coder to explain the available test assertions. Then explain the framework you’ve developed to integrate the testing library with CI, and explain what failures are going to look like. And don’t forget to point them at the documentation you’ve written on the project wiki. This should get your team on board, and now you can start looking for other assertions to write. You can also start tightening the screws on your framework. Make it so that no change can be delivered without at least one test.

Simple DML Checking

The next obvious category of SQL statements to check is simple DML. These are the CRUD verbs: INSERT, SELECT, UPDATE and DELETE. Except that SELECT can be pretty hard to check within SQL, so it’s best to ignore it for now in favor of the others. The problem with DML is knowing what to check for. If you insert a row, should you check that the ROW_COUNT() returns 1? Or should you query for the particular key of that row?

I’m pretty convinced that DML commands in scripts are going to be metadata related. That is, your application may be inserting customer records, or video titles, or whatever you track. But if a developer codes a DELETE, INSERT, or UPDATE into a deployable script, I think the intent is either to perform some kind of bulk fix (convert data fields, or eliminate nulls) or it is adjusting the “background” data — the list of postal codes, state and province names, etc. that your application relies on. In the case of the simple background data, if you added or updated it, you should check it explicitly: assert that for a specific key, the values are as expected.

Some examples of helpful functions would be assert_statement_affects_rowcount(prepared_statement, num_rows), assert_table_column_matches_count('table name', 'column', value, count), and their negatives. The way to do dynamic programming in MySQL is with prepared statements. You can construct a string containing a SQL statement, use PREPARE to assign it to an identifier, and then EXECUTE the identifier:


PREPARE stmt
FROM
CONCAT("SELECT COUNT(*) FROM ", table_name, " WHERE ", field_name, " = ?");



EXECUTE stmt USING value;


In the more complex case of a data format or type update, the update should be encoded in a stored procedure. This way you can unit test the stored procedure with sample data. Once the procedure is ready to go, write a script that calls the procedure – confident already that the procedure will work – and test the invocation by confirming random elements of the data have converted correctly.

Testing Result Sets

As mentioned, testing SELECT can be a real challenge. While it is easy to create an assert_row_count(num) to confirm that your query returned the right number of values, it can be much harder to create a function to assert that a particular ordering exists, or that the results are truly grouped by postal code.

The first thing to do is go ahead and write the assertion functions for number of rows returned by the query. That’s simple, straightforward, and it can stand in place of more complex stuff for a long time. You may not ever get around to writing the more challenging assertions — if the tests are designed well enough, the number of results may be meaningful enough. If you do need to write more, especially if the query is complex, pass the query as a string to your assertion, prepare it as a statement, and make sure the results go into a temporary table. Then you can write whatever horribly complex code you need against the temporary table, without having to worry about losing context.

Testing Triggers

Triggers are going to be harder to test than other pieces of code because of the diverse ways you can invoke them. For example, a DELETE trigger may need to be checked with a simple delete statement as well as with a CASCADE from a foreign key relationship. Beyond creating assertions for the existence and type of triggers (BEFORE/AFTER, INSERT/UPDATE/DELETE, etc.) there isn’t really any obvious set of assertions to write. It’s more a question of deciding how to test the effects of the triggers, and of documenting the ways the trigger can be invoked.

Conclusion

Unit testing, and test-first development, are well established best practices. But it’s easy to overlook the “small stuff” when developers are setting up frameworks for testing their code. Database updates are small, but when they go wrong they can take out the rest of the system. It’s important to apply the same techniques, and get the same value, to this part of your development. Now you can.

The bad thing is that it’s new, and so there are some gotcha’s laying around. I just found one:

I defined a ccnet.config file structured like this:

<cruisecontrol
  xmlns:cb="urn:ccnet.config.builder"
  >
  <!-- Define some local configuration symbols -->
  <cb:define
    FOO="bar"
    />
  <!-- Define a CCnet project template called "cb:PROJECT" -->
  <cb:define
    name="PROJECT"
    >
    <project
      name = "$(projectName)"
      >
      <!--
          guts go here
          -->
    </project>
  </cb>
  <!-- Use template -->
  <cb:PROJECT
    projectName = "My dog has fleas"
    />
</cruisecontrol>

That worked fine, and I could see how to rubber-stamp that project template to all my various continuous-integration areas and other workspaces.

But then I got clever, and decided to separate out the template from the local configuration.

<cruisecontrol
  xmlns:cb="urn:ccnet.config.builder"
  >

  <!-- Define some local configuration symbols -->
  <cb:define
    FOO="bar"
    />

  <!-- Include template(s) file -->
  <cb:include
    href = "project-template.xml"
    />

  <!-- Use template -->
  <cb:PROJECT  projectName = "My dog has fleas"  />
</cruisecontrol>

By unfortunate coincidence, I had to reboot my laptop at this point — I kicked the power strip under the desk, removing power to the docking station. The Lenovo I’m using (client provided) doesn’t handle this at all, so it was restart time. When I came back up, the CruiseControl.NET service wouldn’t start.

Debugging is simple and easy with the “ccnet.exe” command line utility. Just run

ccnet.exe -config:ccnet.config -validate

The error, though, was incomprehensible:

C:\Program Files\CruiseControl.NET\server>ccnet -config:ccnet.config -validate
CruiseControl.NET Server 1.4.0.3400 -- .NET Continuous Integration Server
Copyright c 2008 ThoughtWorks Inc.  All Rights Reserved.
.NET Runtime Version: 2.0.50727.1433    Image Runtime Version: v2.0.50727
OS Version: Microsoft Windows NT 5.1.2600 Service Pack 3        Server locale: e
n-US

[CCNet Server:DEBUG] The trace level is currently set to debug.  This will cause
 CCNet to log at the most verbose level, which is useful for setting up or debug
ging the server.  Once your server is running smoothly, we recommend changing th
is setting in C:\Program Files\CruiseControl.NET\server\ccnet.exe.config to a lo
wer level.
[CCNet Server:INFO] Reading configuration file "C:\Program Files\CruiseControl.NET\server\ccnet.config"
[CCNet Server:ERROR] Exception: The configuration file contains invalid xml: C:\Program Files\CruiseControl.NET\server\ccnet.config
----------
ThoughtWorks.CruiseControl.Core.Config.ConfigurationException: The configuration file contains invalid xml: C:\Program Files\CruiseControl.NET\server\ccnet.config ---> System.Xml.XmlException: 'cb' is an undeclared namespace. Line 1, position 2.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.Throw(String res, String arg, Int32 lineNo, Int32 linePos)
   at System.Xml.XmlTextReaderImpl.LookupNamespace(NodeData node)
   at System.Xml.XmlTextReaderImpl.ElementNamespaceLookup()
   at System.Xml.XmlTextReaderImpl.ParseAttributes()
   at System.Xml.XmlTextReaderImpl.ParseElement()
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XPath.XPathDocument.LoadFromReader(XmlReader reader, XmlSpace space)
   at System.Xml.XPath.XPathDocument..ctor(Stream stream)
   at ThoughtWorks.CruiseControl.Core.Config.Preprocessor.ConfigPreprocessorEnvironment.push_include(String href)
   at ThoughtWorks.CruiseControl.Core.Config.Preprocessor.ConfigPreprocessor.PreProcess(XmlReader input, XmlWriter output, PreprocessorUrlResolver resolver, Uri input_uri)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.CreateXmlValidatingLoader(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.AttemptLoadConfiguration(FileInfo configFile)
   --- End of inner exception stack trace ---
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.AttemptLoadConfiguration(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.Load(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.FileConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.Config.FileWatcherConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.Config.CachingConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.CruiseServer..ctor(IConfigurationService configurationService, ProjectIntegratorListFactory projectIntegratorListFactory, IProjectSerializer projectSerializer)
   at ThoughtWorks.CruiseControl.Core.CruiseServerFactory.Create(Boolean remote, String configFile)
   at ThoughtWorks.CruiseControl.Core.ConsoleRunner.Run()
   at ThoughtWorks.CruiseControl.Console.ConsoleMain.Main(String[] args)
----------

Even worse, the error message is presented in white text on a glaring red background, so as to be all but unreadable. Looking at it now, in retrospect (and black & white), I can see clues that I missed the first time. Specifically, the string “push_include.”

Anyway, the problem turned out to be the way the Configuration Processor deals with XML includes. It is not a C preprocessor — it includes subtrees, not text.

The answer turns out to be in the included file. References in the included file to the “cb:” XML namespace have to be accompanied in that file by some definition. Adding the same xmlns= statement does the trick:

<?xml version="1.0" encoding="UTF-8"?>
<cb:define  xmlns:cb="urn:ccnet.config.builder"  name = "PROJECT"  >
  <!-- blah blah -->
</cb:define>

Problem solved. Now let the rubber-stamping begin!

Review of Guiffy SureMerge

Editorial Notes

Product Explanation

Two-way Comparison

Three-way Comparison

Merging

Directory Compare

Features

Jiff command-line utility

C:\\temp\\Guiffy review\\perl>jiff db2.pl db4.pl
  1  1 open (INPUT, "< $filepageid") || &file_open_error("$filepageid");
  2  2
  3  3 while ($riga=<INPUT>){
  4  4     $nump++;
  5  5     chop($riga);
  6  6     $pagina[$nump] = $riga;
  7  7
  8   <    $sth= $dbh->prepare("SELECT count(*) FROM lognew WHERE
  9   <                         pageid='$pagina[$nump]' and data>='$startdate'");
 10   <    $sth->execute;
 11   <    $totalvisit[$nump] = $sth->fetchrow_array();
     8>    my $totalvisit_sth = $dbh->prepare('SELECT count(*) FROM lognew WHERE
     9>                                        pageid=? and data>=?');
    10>    $totalvisit_sth->execute($pagina[$nump], $startdate);
    11>    $totalvisit[$nump] = $totalvisit_sth->fetchrow_array();
 12 12

    <DIR>     10-Apr-2006 22:48:55 libs
    <DIR>     10-Apr-2006 22:48:45 \\libs\\action
    <DIR>     10-Apr-2006 22:48:45 \\libs\\action\\controller
    <DIR>     10-Apr-2006 22:48:45 \\libs\\action\\controller\\http
    <DIR>     10-Apr-2006 22:48:45 \\libs\\action\\controller\\session
    <DIR>     10-Apr-2006 22:48:45 \\libs\\action\\controller\\templates
    <DIR>     10-Apr-2006 22:48:45 \\libs\\action\\view
    <DIR>     10-Apr-2006 22:48:46 \\libs\\active
    <DIR>     10-Apr-2006 22:50:47 \\libs\\active\\record
<       4,284 10-Apr-2006 22:55:50 \\libs\\active\\record\\Field.php
<          13 10-Apr-2006 22:50:47 \\libs\\active\\record\\NewFile.php
>       4,288 10-Apr-2006 22:56:24 \\libs\\active\\record\\Field.php
    <DIR>     10-Apr-2006 22:48:46 \\libs\\active\\support
    <DIR>     10-Apr-2006 22:48:46 \\libs\\configurator
    <DIR>     10-Apr-2006 22:48:46 \\libs\\context
    <DIR>     10-Apr-2006 22:48:53 \\libs\\creole
    <DIR>     10-Apr-2006 22:48:48 \\libs\\creole\\common
    <DIR>     10-Apr-2006 22:48:48 \\libs\\creole\\contrib
    <DIR>     10-Apr-2006 22:48:53 \\libs\\creole\\drivers
    <DIR>     10-Apr-2006 22:48:49 \\libs\\creole\\drivers\\mssql
    <DIR>     10-Apr-2006 22:48:49 \\libs\\creole\\drivers\\mssql\\metadata
    <DIR>     10-Apr-2006 22:48:50 \\libs\\creole\\drivers\\mysql
    <DIR>     10-Apr-2006 22:48:50 \\libs\\creole\\drivers\\mysql\\metadata
    <DIR>     10-Apr-2006 22:48:51 \\libs\\creole\\drivers\\mysqli

SureMerge command-line utility

<<<<<<< C:\\mydocuments\\Sources\\php data layer\\med2\\libs\\active\\record\\Field.php
class Field extends Object implements Persistable {
=======
class Field extends Object 
	implements Persistant 
{
>>>>>>> c:\\mydocuments\\Sources\\php data layer\\med3\\libs\\active\\record\\Field.php

HTML comparison reports

International Character Encodings

C:\\MyDocuments\\Sources\\Php Data Layer>sed -ne 41p field.uni.txt|od -t x1z
0000000 00 63 00 6c 00 61 00 73 00 73 00 20 00 46 00 69  >.c.l.a.s.s. .F.i<
0000020 00 65 00 6c 00 64 00 20 00 65 00 78 00 74 00 65  >.e.l.d. .e.x.t.e<
0000040 00 6e 00 64 00 73 00 20 00 4f 00 62 00 6a 00 65  >.n.d.s. .O.b.j.e<
0000060 00 63 00 74 00 20 00 69 00 6d 00 70 00 6c 00 65  >.c.t. .i.m.p.l.e<
0000100 00 6d 00 65 00 6e 00 74 00 73 00 20 00 50 00 65  >.m.e.n.t.s. .P.e<
0000120 00 72 00 73 00 69 00 73 00 74 00 61 00 62 00 6c  >.r.s.i.s.t.a.b.l<
0000140 00 65 00 20 00 7b 00 0d 00 0a                    >.e. .{....<
0000152

C:\\MyDocuments\\Sources\\Php Data Layer>sed -ne 41p field.php|od -t x1z
0000000 63 6c 61 73 73 20 46 69 65 6c 64 20 65 78 74 65  >class Field exte<
0000020 6e 64 73 20 42 61 73 65 20 69 6d 70 6c 65 6d 65  >nds Base impleme<
0000040 6e 74 73 20 50 65 72 73 69 73 74 61 62 6c 65 20  >nts Persistable <
0000060 7b 0a                                            >{.<
0000062

C:\\MyDocuments\\Sources\\Php Data Layer>head -1 diff.html | od -t x1z
0000000 fe ff 00 3c 00 21 00 44 00 4f 00 43 00 54 00 59  >...<.!.D.O.C.T.Y<
0000020 00 50 00 45 00 20 00 48 00 54 00 4d 00 4c 00 20  >.P.E. .H.T.M.L. <
0000040 00 50 00 55 00 42 00 4c 00 49 00 43 00 20 00 22  >.P.U.B.L.I.C. ."<
0000060 00 2d 00 2f 00 2f 00 57 00 33 00 43 00 2f 00 2f  >.-././.W.3.C././<
0000100 00 44 00 54 00 44 00 20 00 48 00 54 00 4d 00 4c  >.D.T.D. .H.T.M.L<
0000120 00 20 00 34 00 2e 00 30 00 31 00 20 00 54 00 72  >. .4...0.1. .T.r<
0000140 00 61 00 6e 00 73 00 69 00 74 00 69 00 6f 00 6e  >.a.n.s.i.t.i.o.n<
0000160 00 61 00 6c 00 2f 00 2f 00 45 00 4e 00 22 00 20  >.a.l././.E.N.". <
0000200 00 22 00 68 00 74 00 74 00 70 00 3a 00 2f 00 2f  >.".h.t.t.p.:././<
0000220 00 77 00 77 00 77 00 2e 00 77 00 33 00 2e 00 6f  >.w.w.w...w.3...o<
0000240 00 72 00 67 00 2f 00 54 00 52 00 2f 00 68 00 74  >.r.g./.T.R./.h.t<
0000260 00 6d 00 6c 00 34 00 2f 00 6c 00 6f 00 6f 00 73  >.m.l.4./.l.o.o.s<
0000300 00 65 00 2e 00 64 00 74 00 64 00 22 00 3e 00 0d  >.e...d.t.d.".>..<
0000320 00 0a   

Java API

Support

Accuracy

Performance

Ease of Use

Effectiveness

Cost

Conclusion

Guiffy SureMerge

Compare (diff) Tools