Doing better » Tools

Using Doxygen’s \test command with C++

Austin Hastings — Sat, 28 Aug 2010 21:19:39 +0000

I’m working on some C++ code that is documented using Doxygen. Nothing earth-shattering there.

But I’m doing unit testing, and writing unit tests. In this case, I’m using the boost C++ libraries. That means my tests don’t look like classes, the way they might look if I was using CppUnit or CxxUnit. Instead, they look like macros:
BOOST_AUTO_TEST_CASE( null_ctor ) {

Arena & sut = Simple::Arena(); BOOST_CHECK( sut.capacity() != 0 ); }

The macro expansion is really quite clever, in a Lovecraftian “there are secrets man was not meant to know” fashion. And reading them has certainly improved my knowledge of C++. (But it also permanently lowered my SAN by a few points, I think.)

Anyway, Doxygen offers this command called \test that seems tailor-made for testing.

Well, it’s not. The command description is very “vague,” in that you use it to add entries (paragraphs) to the “test list.”

Some experimentation will show that the apparent use is to document something (a class, a function) and add a bunch of these \test entries, just like adding \param commands. And the entity you are documenting will then have a Test: section in its documentation.

This is totally not what I want. And I’m pretty sure it’s not what anybody doing TDD wants. What I want is to somehow tie my tests, which are in a separate location (different file, different class, different namespace, etc.), back to the class or function that I’m testing.

The best way I’ve found so far to accomplish this is to lie to Doxygen about what is being documented. Here’s my current scheme:
#define BOOST_TEST_DYN_LINK #include


#include 
using namespace Bronze::Memory;
BOOST_AUTO_TEST_SUITE( Memory_SimpleArena )
/// \class Bronze::Memory::Simple::Arena

/// \test \b null_ctor Confims that a SimpleArena no-args construction will

/// use the default size, whatever that is. (1m)
BOOST_AUTO_TEST_CASE( null_ctor ) {

Arena & sut = Simple::Arena(); BOOST_CHECK( sut.capacity() != 0 ); }

Adding the \class command tells Doxygen to re-open the documentation of the target class. The \test command is then associated with the “right” class, and I include the testcase name in bold as part of the test description.

This still doesn’t generate a link to the right source file (containing the test), and it doesn’t really know anything about the test case. So it’s not a perfect solution. But it does enable me to put my testcase documentation with my testcases, while having the documentation actually show up attached to the right class.

Which ‘which’ is which?

Austin Hastings — Tue, 10 Aug 2010 22:13:27 +0000

The ‘which’ utility is one of those really useful commands that never seems to cross the bridge from Unix to Windows. The CMD.EXE special %$PATH:f syntax seems to promise some relief, but of course it’s never that simple – I at least want to type “which foo” rather than “which foo.exe”.

So here’s which.cmd – a script that tries to DWIW.

Note that this script is wrong in one key detail: it searches inside-out. That is, any “foo.COM” will take precedence over any “foo.EXE” even if the .exe version occurs earlier in the PATH.

@echo off
REM Copyright (c) 2010, Austin Hastings.
REM This file may be used for any purpose without restriction.

REM NOTE: This script does not handle multiple entries with different
REM extensions correctly. It returns the first matching EXTENSION, rather
REM than returning the first DIRECTORY with any EXTENSION.

REM Analogous to unix 'which' command, look for a matching runnable
REM in %PATH% and print the location.

SETLOCAL ENABLEDELAYEDEXPANSION

set check_extensions= ;%PATHEXT%

:ext_loop
if "%check_extensions%" == "" goto done

for /F "tokens=1* delims=;" %%E in ( "%check_extensions%" ) do (
	set check_extensions=%%F
	set target=%1%%E
	for %%W in (  !target! ) do set answer=%%~f$PATH:W
)

if "%answer%" == "" goto ext_loop

:done
if "%answer%" == "" goto not_found

@echo %answer%
goto exit

:not_found
@echo No runnable matching '%1' was found.

:exit

Some bash goodness

Austin Hastings — Wed, 28 Oct 2009 15:09:48 +0000

Here’s some bash goodness (well, not really) to make ‘less’ a little bit more useful.

less() {

local -a args

for arg

case “$arg” in

*:[[:digit:]]* )

line=${arg/#+(?):/}

args[${#args[*]}]=”+$line”

arg=${arg/%:+(?)/}

;;

esac

args[${#args[*]}]=$arg

done

$( which less ) “${args[@]}”

}

Basically, this shell function loads when you tell it to (in your .profile, likely), and it replaces the ‘less’ command. When you type ‘less …’ on the command line, the function runs.

It scans through the args, looking for one like filename.c:24, and if it finds that kind of arg, it replaces it with +24 filename.c — translating the syntax used by a *lot* of compilers into the syntax used by less for opening a file and jumping directly to the line number.

Unit tests for MySQL scripts

Austin Hastings — Sun, 26 Apr 2009 01:07:09 +0000

Recently I had the opportunity to develop a unit testing framework for MySQL scripts in mostly-pure SQL. While I can’t share the code, I can certainly describe what we did, and why we did it. Hopefully it will be useful to you.

My client was reorganizing a development team into an agile mode in order to deal with an outsourcing disaster. It was one of those horror stories that starts out like “We were 3 years in to a 10 month project, when …” I was brought on to help ramp up their build and deployment processes, to get continuous integration and testing systems up, and to help mechanize their deployments.

The impetus for the MySQL unit test framework came from a short term contractor. The one “constant” in the otherwise highly-variable environment had been that the team wasn’t changing. Suddenly, that stopped being true. We had a handful of contractors in for a short term surge, and none of them were familiar with the application, or with SQL (it seemed). The offender was using the CI server as his syntax checker for database scripts. Yikes!

Luckily, the garbage he delivered was syntactically invalid. The mysql command-line client rejected the script and exited with an error status, which caused the CI script to log a failure, and … voila! Everyone credited me with doing a great job of implementing CI, because it was even checking the database stuff. Like a good consultant, I said “Yes. Of course.” Then I ran back to my desk, thinking “Oh, no! I’ve got to come up with a unit test framework for the MySQL stuff, pronto!” And I did.

Solutions

The software was a pretty complex web app, and the target environment was a mixture of Linux and Windows. As a result, when I looked at existing SQL unit test frameworks (dbUnit and sqlUnit, respectively) I had to reject them. They are decent products – and free – but the fact is that they expect certain things about their runtime environment that we couldn’t deliver. We had to build our own MySQL unit test framework, ideally in “pure” MySQL.

Fortunately for me, a nice guy named Giuseppe Maxia maintains a web site at datacharmer.org, and he has written a library of general-purpose MySQL code. (Called the General Purpose Stored Routine library, or gp_sr_lib.) Among the all this well-written, free, already debugged code was a set of testing routines. Thirty minutes into my development effort, I was already 80% done. Woo-hoo!

The GP library test routines are a good start on a mysqlUnit library, and I strongly recommend that if you need such a library you stop reading here and go download the code. Right now.

Rolling (y)our Own

There are a couple of gotchas that you’ll want to watch out for in your own unit testing library. First, the tests should be able to integrate into your CI engine. In particular, you need to decide on and implement some kind of failure mode – either fail when the first problem occurs, or fail after running all test cases. If you’re running the mysql client via some kind of exec call, be aware that while syntax errors will cause the client to exit with a failure code, other errors will not cause a failure. Instead, the client prints a diagnostic and returns zero. What’s more, the syntax does not support a deliberate abort from within the SQL engine.

In my own case, I chose to follow the model that Maxia has provided, and NOT fail as soon as a test fails. Rather, my code collected the results of all the tests that were run, and summarized them. In order to get “fail on error” behavior out of the mysql command line client, I coded a routine that would perform a “SELECT force_mysql_client_abort FROM no_such_table”. This definitely does cause the client to abort, at least until some developer creates a table called “no_such_table.” Then I wrapped the test cases in an “ignore failure” script that called the summarizer after all the cases had run, which in turn would call the abort routine if the error count was non-zero.

Another thing I chose to “fix” was the routine names. Maxia’s code includes both functions and procedures, but the names are not really in keeping with the style used in the various xUnit test frameworks. The library contains a procedure for asserting the existence of a database table. It is called as:

CALL check_table('db name', 'table name');

I opted to “embrace and extend” the library. I wrapped these types of checks into functions, like this:

set_database('db name'); assert_table_exists('table name'); assert_table_exists_in_db('other table', 'other db');

And in general, wherever there is a straight-ahead assert, I also added an “assert_not”. In terms of effect, there is no real difference in behavior between Maxia’s naming and my own. But I felt that the developers, who were using jUnit and nUnit, would be more comfortable learning a set of routines that had a similar structure and style.

Going Most of the Way: DDL

I tried to make a list of the various things that SQL scripts might do. The most obvious — and this probably accounted for two thirds or more of the scripts we had — is DDL statements. DDL (Data Definition Language) is that subset of SQL that relates to defining databases, tables, indexes, triggers, and the like. While the most frequently executed SQL statements are DML (Data Manipulation Language), DDL is where it’s at for most of the human-generated script files: CREATE TABLE, ADD INDEX, etc.

The great thing is that the assertions for these statements are trivial to write. Maxia’s library includes the CHECK_TABLE procedure, and some code for looking up procedures and functions. Adding other stuff, like keys and constraints, is pretty straightforward. By the end of the day you should have a long list of functions like assert_table_exists, assert_index_exists, assert_database_exists, and assert_column_has_type.

Deploy Early

At this point, you’re ready to go live. Except that you need to build a framework to run these tests. That’s going to depend a lot on your deployment process, and on your particular CI environment. Sorry. The simplest idea would be to write a script that matches the file names of test files, and runs them one at a time. Call that script from CI, and make sure the exit status is meaningful.

Forget the next couple of sections, because most of the database changes you’re likely to see are DDL. Developers and DBAs do a lot of adding fields, adding indexes, and sometimes adding tables. If you have just the existence assertions for each kind of object, your library will cover two-thirds or more of the statements being delivered. That’s not a trivial amount of test coverage for a day or two of work. Ship it! Your developers will need some training, and you’ll need to beat on the thing once it’s deployed to your CI servers to make sure that the errors are coming out correctly.

Sell to the Development Team

I recommend that you write some simple tests of things you know to be true. Confirm the positive cases first. Then sit down with a smart, involved developer and pair develop some unit tests for code he hasn’t written yet. Do it the way you want the developers to do it, including deciding how to handle assertions that don’t exist yet.

In our case, the DBAs had already established a naming convention for database update files. I chose to duplicate the file names, which would look like ####_description.sql, changing the .sql extension to .test.sql. This let the testing framework associate the test with the delivered change. When running a “from zero” deployment, the test framework would then run TEST-CHANGE-TEST and expect the first test to fail and the second to pass. This demonstrated that the test was meaningful for the change (remember, the change is supposed to repair a shortcoming that is demonstrated by the test).

Once you’ve got buy-in from your chosen developer, sit down and develop a presentation that the two of you can give. Point to specific examples of blown deployments as reasons for unit testing the SQL. Then get the coder to explain the available test assertions. Then explain the framework you’ve developed to integrate the testing library with CI, and explain what failures are going to look like. And don’t forget to point them at the documentation you’ve written on the project wiki. This should get your team on board, and now you can start looking for other assertions to write. You can also start tightening the screws on your framework. Make it so that no change can be delivered without at least one test.

Simple DML Checking

The next obvious category of SQL statements to check is simple DML. These are the CRUD verbs: INSERT, SELECT, UPDATE and DELETE. Except that SELECT can be pretty hard to check within SQL, so it’s best to ignore it for now in favor of the others. The problem with DML is knowing what to check for. If you insert a row, should you check that the ROW_COUNT() returns 1? Or should you query for the particular key of that row?

I’m pretty convinced that DML commands in scripts are going to be metadata related. That is, your application may be inserting customer records, or video titles, or whatever you track. But if a developer codes a DELETE, INSERT, or UPDATE into a deployable script, I think the intent is either to perform some kind of bulk fix (convert data fields, or eliminate nulls) or it is adjusting the “background” data — the list of postal codes, state and province names, etc. that your application relies on. In the case of the simple background data, if you added or updated it, you should check it explicitly: assert that for a specific key, the values are as expected.

Some examples of helpful functions would be assert_statement_affects_rowcount(prepared_statement, num_rows), assert_table_column_matches_count('table name', 'column', value, count), and their negatives. The way to do dynamic programming in MySQL is with prepared statements. You can construct a string containing a SQL statement, use PREPARE to assign it to an identifier, and then EXECUTE the identifier:

PREPARE stmt FROM CONCAT("SELECT COUNT(*) FROM ", table_name, " WHERE ", field_name, " = ?");

EXECUTE stmt USING value;

In the more complex case of a data format or type update, the update should be encoded in a stored procedure. This way you can unit test the stored procedure with sample data. Once the procedure is ready to go, write a script that calls the procedure – confident already that the procedure will work – and test the invocation by confirming random elements of the data have converted correctly.

Testing Result Sets

As mentioned, testing SELECT can be a real challenge. While it is easy to create an assert_row_count(num) to confirm that your query returned the right number of values, it can be much harder to create a function to assert that a particular ordering exists, or that the results are truly grouped by postal code.

The first thing to do is go ahead and write the assertion functions for number of rows returned by the query. That’s simple, straightforward, and it can stand in place of more complex stuff for a long time. You may not ever get around to writing the more challenging assertions — if the tests are designed well enough, the number of results may be meaningful enough. If you do need to write more, especially if the query is complex, pass the query as a string to your assertion, prepare it as a statement, and make sure the results go into a temporary table. Then you can write whatever horribly complex code you need against the temporary table, without having to worry about losing context.

Testing Triggers

Triggers are going to be harder to test than other pieces of code because of the diverse ways you can invoke them. For example, a DELETE trigger may need to be checked with a simple delete statement as well as with a CASCADE from a foreign key relationship. Beyond creating assertions for the existence and type of triggers (BEFORE/AFTER, INSERT/UPDATE/DELETE, etc.) there isn’t really any obvious set of assertions to write. It’s more a question of deciding how to test the effects of the triggers, and of documenting the ways the trigger can be invoked.

Conclusion

Unit testing, and test-first development, are well established best practices. But it’s easy to overlook the “small stuff” when developers are setting up frameworks for testing their code. Database updates are small, but when they go wrong they can take out the rest of the system. It’s important to apply the same techniques, and get the same value, to this part of your development. Now you can.

Gotcha: XML include processing in CruiseControl.NET

Austin Hastings — Tue, 24 Jun 2008 16:24:36 +0000

Release 1.4 of CruiseControl.NET includes a new Configuration Preprocessor. The great thing is that this provides a C preprocessor-like macro syntax, so projects can be made using relatively powerful templates.

The bad thing is that it’s new, and so there are some gotcha’s laying around. I just found one:

I defined a ccnet.config file structured like this:

  
  
  
  
    
      
    
  
  
  

That worked fine, and I could see how to rubber-stamp that project template to all my various continuous-integration areas and other workspaces.

But then I got clever, and decided to separate out the template from the local configuration.

  
  

  
  

  
  

By unfortunate coincidence, I had to reboot my laptop at this point — I kicked the power strip under the desk, removing power to the docking station. The Lenovo I’m using (client provided) doesn’t handle this at all, so it was restart time. When I came back up, the CruiseControl.NET service wouldn’t start.

Debugging is simple and easy with the “ccnet.exe” command line utility. Just run
ccnet.exe -config:ccnet.config -validate

The error, though, was incomprehensible:

C:\Program Files\CruiseControl.NET\server>ccnet -config:ccnet.config -validate
CruiseControl.NET Server 1.4.0.3400 -- .NET Continuous Integration Server
Copyright c 2008 ThoughtWorks Inc.  All Rights Reserved.
.NET Runtime Version: 2.0.50727.1433    Image Runtime Version: v2.0.50727
OS Version: Microsoft Windows NT 5.1.2600 Service Pack 3        Server locale: e
n-US

[CCNet Server:DEBUG] The trace level is currently set to debug.  This will cause
 CCNet to log at the most verbose level, which is useful for setting up or debug
ging the server.  Once your server is running smoothly, we recommend changing th
is setting in C:\Program Files\CruiseControl.NET\server\ccnet.exe.config to a lo
wer level.
[CCNet Server:INFO] Reading configuration file "C:\Program Files\CruiseControl.NET\server\ccnet.config"
[CCNet Server:ERROR] Exception: The configuration file contains invalid xml: C:\Program Files\CruiseControl.NET\server\ccnet.config
----------
ThoughtWorks.CruiseControl.Core.Config.ConfigurationException: The configuration file contains invalid xml: C:\Program Files\CruiseControl.NET\server\ccnet.config ---> System.Xml.XmlException: 'cb' is an undeclared namespace. Line 1, position 2.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.Throw(String res, String arg, Int32 lineNo, Int32 linePos)
   at System.Xml.XmlTextReaderImpl.LookupNamespace(NodeData node)
   at System.Xml.XmlTextReaderImpl.ElementNamespaceLookup()
   at System.Xml.XmlTextReaderImpl.ParseAttributes()
   at System.Xml.XmlTextReaderImpl.ParseElement()
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XPath.XPathDocument.LoadFromReader(XmlReader reader, XmlSpace space)
   at System.Xml.XPath.XPathDocument..ctor(Stream stream)
   at ThoughtWorks.CruiseControl.Core.Config.Preprocessor.ConfigPreprocessorEnvironment.push_include(String href)
   at ThoughtWorks.CruiseControl.Core.Config.Preprocessor.ConfigPreprocessor.PreProcess(XmlReader input, XmlWriter output, PreprocessorUrlResolver resolver, Uri input_uri)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.CreateXmlValidatingLoader(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.AttemptLoadConfiguration(FileInfo configFile)
   --- End of inner exception stack trace ---
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.AttemptLoadConfiguration(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.Load(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.FileConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.Config.FileWatcherConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.Config.CachingConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.CruiseServer..ctor(IConfigurationService configurationService, ProjectIntegratorListFactory projectIntegratorListFactory, IProjectSerializer projectSerializer)
   at ThoughtWorks.CruiseControl.Core.CruiseServerFactory.Create(Boolean remote, String configFile)
   at ThoughtWorks.CruiseControl.Core.ConsoleRunner.Run()
   at ThoughtWorks.CruiseControl.Console.ConsoleMain.Main(String[] args)
----------

Even worse, the error message is presented in white text on a glaring red background, so as to be all but unreadable. Looking at it now, in retrospect (and black & white), I can see clues that I missed the first time. Specifically, the string “push_include.”

Anyway, the problem turned out to be the way the Configuration Processor deals with XML includes. It is not a C preprocessor — it includes subtrees, not text.

The answer turns out to be in the included file. References in the included file to the “cb:” XML namespace have to be accompanied in that file by some definition. Adding the same xmlns= statement does the trick:



xmlns:cb="urn:ccnet.config.builder"  name = "PROJECT"  >

Problem solved. Now let the rubber-stamping begin!

[wiki | reviews:byname:guiffy_suremerge:20060706]

Austin Hastings — Sat, 19 Apr 2008 01:15:50 +0000

Review of Guiffy SureMerge

Editorial Notes

Date: March 2006
Product: Guiffy SureMerge 7.2 “Dogwood”
Vendor: Guiffy Software (http://www.guiffy.com)
Reviewer: Austin Hastings

Bill Ritcher, coder and CEO of Guiffy, is aware of the review. He became aware about half-way through when I was contacting him for help with various issues. After the review was published, Guiffy began providing a link to the review from their product web site.

Product Explanation

Guiffy (pronounced “Goofy” according to owner/developer Bill Ritcher) SureMerge is a java-based, platform-neutral visual compare and merge utility for software and content developers. It is compatible with Java 5 (as well as earlier releases) and is available for OpenVMS and OS/2, as well as less-popular operating systems like Linux, Windows, and Mac OS/X.

Most of the time, comparison and merge tools are listed as being tools for developers. In reality, though, this is not true. Comparison and merge tools are tools for leads, build managers, and SCMs—the folks who get called when something goes wrong. But we all wish that developers would fix the problems they introduce, so most companies distribute these tools to everyone. SureMerge supports developers with “the usual” range of compare and merge features, discussed below. In addition, it contains features that will be near and dear to the heart of build managers, team leads, and change managers everywhere—tools that make it easy to deal with whole development teams and whole trees of files.

The product comes with a collection of executables including stand-alone compare and merge command line tools. One of those programs is a command-line merge utility called suremerge. The GUI version is named guiffy. I will refer to them by name when it is relevant. Otherwise I will call the whole software bundle SureMerge, and the company Guiffy.

Two-way Comparison

Two-way comparison of files is what comes to mind immediately when you think about comparisons: put two files next to each other and see where the changes are. Most readers will be familiar with the diff and sdiff commands for comparing files and displaying the results. A graphical rendition is much clearer and easier to understand.

When you evaluate SureMerge, the Guiffy web site (http://www.guiffy.com) includes a white paper with a series of test cases. The white paper highlights different kinds of changes and how they are handled by merge & compare tools, using Guiffy source code in the examples. This screen shot was taken using one of the included test cases. (No, I don't have access to Guiffy's source code.)

A very nice feature of the guiffy GUI 1) is character-by-character highlighting of differences. If you are using a language that tends to have 'dense' statements, this can be a life-saver, highlighting where a full stop ('.') changed to a comma or where a single quote became a back tick. Be careful with character-by-character differencing: like all compare algorithms, it is prone to look for similarities that may not be there:

The example shows two random DNA sequences. I used perl to randomly emit the four letters A, C, G, T. As you can see, guiffy was willing to “walk a mile” to show me the exact changes, when in fact I just made up two totally different sequences. 2)

That was a dumb example made to demonstrate a point: guiffy offers a configuration setting that lets you determine how much change is “too much”. A better example is refactoring code. Consider this change, taken from Michael Schwern's excellent Perl refactoring tutorial:

This example also has character highlighting turned on, so what happened? The answer lies in a configuration dialog, Options>Show:

You can choose to show difference that are up to 100% of the line, which is what I did for the DNA sequence above. The default is 50%, as shown, but when you are merging source code the devil is in the details. In the example above, line 9 changed by more than 50% because of the long variable names. As a result, it was “too much change” to highlight. With a single buttonpress, a mouseclick, and 'Enter' I have a different picture:

This is how the same code looks at a 75% change cutoff. Suddenly it's clear that the perl variables have been pulled out of the SQL statement and replaced by bind variables. And the variables are now passed to $sth→execute as method args. This is great stuff! Just to be a little nit-picky, I'm going to suggest that Guiffy make the Show Options page into a toolbar control or some kind of dockable: when you are looking at differences in code it can be important to quickly toggle back and forth.

Three-way Comparison

Beyond two-way comparison lies three-way. (Relax: three is the limit!) With three-way comparison, and three-way merging, you are comparing a single common ancestor with two different changed files. This is the tool to use when two developers have made different changes to the same file, and you're stuck merging them together. Sound familiar?

In the illustration (below) you can see how guiffy presents two changes that conflict with each other. In this case, different developers have allocated the constant value 7 to different purposes: the names CWP and ENA.

To merge the changes it isn't sufficient to copy both new lines into the resulting file. Instead, one line will have to be manually edited to convert its value from 7 to 8. The tool compares both changed files against the parent, and then compares the change sets. For each part of the file, the possibilities are:

Parent vs. Me	Other vs. Me	Other vs. Parent	Interpretation
Same	Same	Same	Unchanged in both branches.
Same	Different	Different	Changed only in other branch. Take that change.
Different	Same	Different	Changed identically in both branches. Take this change.
Different	Different	Same	Changed only in this branch. Take this change.
Different	Different	Different	Changed in both branches. Merge these changes by hand.

Nearly all merge programs use this kind of logic to determine how to respond to changes. There are a couple of potential problems, however. First comes finding the smallest region in the file that constitutes a single change. We've all seen cases where a compare program finds a single closing curly brace that is “equal” to similar text in the other file and so declares that point the end of one change and the start of another. Finding the boundaries of a “single” change can be challenging.

Some cases aren't so clear-cut. For example, what if two developers insert code in the same location in the file, as illustrated above. You or I can tell it's a conflict (two allocations of the number 7), but some tools won't, since the two changes can be automatically merged by inserting both! The white paper I mentioned above has 5 simple scenarios for compare & merge. I was surprised to learn that every other tool in the market gets at least one of those scenarios wrong! The cases are simple, and source code is provided: try them yourself.

Knowing what to ignore can be an issue, too. If the files in question have been translated from one operating system to another, or edited with different tools, things like white space and line terminators may be different. Having the ability to ignore these trivial changes is important. SureMerge has the ability, in spades. You can ignore whitespace at the beginning, at the end, or everywhere in the file. You can ignore changes matching a regex, or changes in a certain range of columns. The only thing SureMerge won't ignore for you is phone calls from your boss.

Finally, there is the risk of an identical-but-wrong change. In this scenario, possibly caused by out-of-band sharing, one part of a change might be present in both child versions but be fatal to one of the changes. Typically, you'll see this when the two developers have discussed the changes but haven't explicitly thought about each change. There's no way to catch these mechanically—your best bet is good unit testing.

guiffy does offer a nice feature that comes close to being psychic when these changes happen within a few lines: it reports an 'attention' on code changes that are not in conflict, but are close together. If two developers made changes just a few lines apart, the software knows that it could merge them automatically, but is smart enough to say, “Hey! Somebody should check this.” Just like the Federal Aviation Administration, it considers even a near miss to be something important. Very nice.

Merging

As you might expect from a product named SureMerge, merging is a strong feature. There are a lot of different ways you can “merge” code via the GUI:

Just start typing. If you are using guiffy in merge mode, type (or copy & paste) what you want into the “merged” window—guiffy won't let you change either of the original inputs. If you are using compare mode, you can edit in either file just like a text editor.
Use Two-way merging. If all you have is two different files, you can choose between alternatives: Of course, you can also just type: sometimes the merge process isn't about selecting between two alternatives, but combining them. If you need to type, or copy and paste, to get the results you want, just point and click.
Use Three-way merging. As mentioned above, guiffy is excellent at determining just exactly which changes can be merged automatically, and which changes really do require the attention of a human. Of the five scenarios described above, only 1 requires a human to help with the work. In my experience, this reflects reality fairly well: I'd guess that the number of 'real world' merge steps requiring human attention is somewhere between 5 and 30 percent, depending on your code.

Regardless, guiffy gives you the same basic interface (the 'parent' is used to compute the changes, but isn't shown) as with two-way merging, and lets you do the same things: select code from either source as a starting point, then edit the code to the result you want.

Directory Compare

Another nice feature of guiffy is the directory tree compare GUI. This uses the same color scheme as the file compare to show you files that have been added, deleted, or changed between two directory trees:

This is a great place to start when you are trying to understand what has happened to your code. Seeing the higher-level view of what has been changed lets you organize your approach to understanding and/or merging the changes. The interface is visually clean and attractive, as well as intuitive. The obvious tools to fix to directory tree issues—copying, renaming, and and deleting files—are supported in the tree view. I do have a few gripes about this interface, however:

First, there is no indication at the directory level that children have changed. For a deep source tree this may make it harder to grasp the whole impact of a change. Mitigating this is the fact that the GUI takes you right to the first change, and you can use the next/previous change buttons to navigate.
Second, guiffy doesn't do the “obvious” thing when dealing with trees. If I invoke guiffy with three or four directory names, as I would do for merging files, the tool doesn't automatically create the new “merged” directory tree. Neither does it correctly populate the “parent” field when I try to SureMerge a changed file.

The former is evidence of how addictive and intuitive the rest of the product is: of course it will work this way—just bring up the directories and start resolving conflicts! It actually took me a while to grasp that I was wrong. The latter behavior—incorrectly populating the parent field—seems to be just a bug. guiffy remembers the last value I used, instead of populating it from the command line.

Third, there is no good way to move files from one directory to another: the 'move' operation is an extended version of 'rename'. I suspect this is a drag & drop problem—there are some other issues with drag & drop discussed later. For Java development in particular, this may be a problem when refactoring. (Or not: much refactoring will involve opening the 'rename' dialog anyway—maybe combining move with rename won't be a problem.)

Features

In addition to fast, accurate 2- and 3-way merging, SureMerge includes

jiff – a command-line compare utility

suremerge – a command-line merge utility

Recursive comparison of directory hierarchies in guiffy and jiff

HTML comparison reports in guiffy

Support for international character encodings

A Java API for integrating SureMerge functionality into your own programs.

Really responsive support

There's a wealth of other features listed at http://www.guiffy.com but these stand out as being important for automation and build/team management.

Jiff command-line utility

jiff is not diff. Instead, it is a command-line tool for showing differences. It does not use the diff algorithms (a feature: Guiffy claims their algorithms are better), it does not produce the eleventy-four different specialized formats for feeding to other tools. It does not use the same command-line switches: another feature, since diff does not have a switch for specifying the character encoding of the input files, while jiff does.

What jiff does is produce a very readable, very comprehensible text version of the results shown by the guiffy GUI. Here's an example:

C:\\temp\\Guiffy review\\perl>jiff db2.pl db4.pl
  1  1 open (INPUT, "< $filepageid") || &file_open_error("$filepageid");
  2  2
  3  3 while ($riga=){
  4  4     $nump++;
  5  5     chop($riga);
  6  6     $pagina[$nump] = $riga;
  7  7
  8   <    $sth= $dbh->prepare("SELECT count(*) FROM lognew WHERE
  9   <                         pageid='$pagina[$nump]' and data>='$startdate'");
 10   <    $sth->execute;
 11   <    $totalvisit[$nump] = $sth->fetchrow_array();
     8>    my $totalvisit_sth = $dbh->prepare('SELECT count(*) FROM lognew WHERE
     9>                                        pageid=? and data>=?');
    10>    $totalvisit_sth->execute($pagina[$nump], $startdate);
    11>    $totalvisit[$nump] = $totalvisit_sth->fetchrow_array();
 12 12

jiff offers a directory-compare behavior similar to guiffy, but I had a few problems with this.

First, I expected jiff -r to behave just like diff -r: that was arguably my problem. Instead, jiff -r does just a tree comparison: it does not show file differences as it goes.

Second, getting jiff -r to show files that have changed between the two trees requires specifying the -bxt (Verify files match) option: this one even stumped the author for a while. 3)

Third, the option syntax is awkward: -diffs shows “only the differences” for files, but -nomats shows “no matching files” for folder comparisons.

Finally, the folder compare wants to show the names of folders it has checked, even if there were no differences: -nomats didn't prevent it from burying my file changes amid a maze of directory names, all alike:

         10-Apr-2006 22:48:55 libs
         10-Apr-2006 22:48:45 \\libs\\action
         10-Apr-2006 22:48:45 \\libs\\action\\controller
         10-Apr-2006 22:48:45 \\libs\\action\\controller\\http
         10-Apr-2006 22:48:45 \\libs\\action\\controller\\session
         10-Apr-2006 22:48:45 \\libs\\action\\controller\\templates
         10-Apr-2006 22:48:45 \\libs\\action\\view
         10-Apr-2006 22:48:46 \\libs\\active
         10-Apr-2006 22:50:47 \\libs\\active\\record
<       4,284 10-Apr-2006 22:55:50 \\libs\\active\\record\\Field.php
<          13 10-Apr-2006 22:50:47 \\libs\\active\\record\\NewFile.php
>       4,288 10-Apr-2006 22:56:24 \\libs\\active\\record\\Field.php
         10-Apr-2006 22:48:46 \\libs\\active\\support
         10-Apr-2006 22:48:46 \\libs\\configurator
         10-Apr-2006 22:48:46 \\libs\\context
         10-Apr-2006 22:48:53 \\libs\\creole
         10-Apr-2006 22:48:48 \\libs\\creole\\common
         10-Apr-2006 22:48:48 \\libs\\creole\\contrib
         10-Apr-2006 22:48:53 \\libs\\creole\\drivers
         10-Apr-2006 22:48:49 \\libs\\creole\\drivers\\mssql
         10-Apr-2006 22:48:49 \\libs\\creole\\drivers\\mssql\\metadata
         10-Apr-2006 22:48:50 \\libs\\creole\\drivers\\mysql
         10-Apr-2006 22:48:50 \\libs\\creole\\drivers\\mysql\\metadata
         10-Apr-2006 22:48:51 \\libs\\creole\\drivers\\mysqli

SureMerge command-line utility

Where jiff would accept a relative path on the command line, like jiff med2\\libs\\active\\record\\field.php med3\\libs\\active\\record\\field.php suremerge would not. Needless to say, suremerge and I did not get off to a great start.

In addition, it has problems parsing directory names that include spaces on Windows. When I specified a directory with a leading quote, as “C:\\MyDocuments\\Sources\\Php Data Layer”\\med2\\… all was well. If I tried to use quotes inside the directory, as C:\\MyDocuments\\Sources\\“Php Data Layer”\\med2\\… then I got a usage statement, and eventually a sore forehead from banging it against my desk. This is documented in the help text, but it's just a little bit different from common Windows behavior.

Apparently, suremerge makes no use whatsoever of the current directory. It does not resolve relative paths against it, and it will not use it to resolve its output file, either. Once you overcome these challenges, though, suremerge does just exactly what you expected it would do: it performs the same merges from the command line that guiffy does from the GUI. The remaining problem, of course, is when lines conflict with each other. Anyone who has used CVS will be familiar with the result:

<<<<<<< C:\\mydocuments\\Sources\\php data layer\\med2\\libs\\active\\record\\Field.php
class Field extends Object implements Persistable {
=======
class Field extends Object 
	implements Persistant 
{
>>>>>>> c:\\mydocuments\\Sources\\php data layer\\med3\\libs\\active\\record\\Field.php

Because of Guiffy's improved algorithms, suremerge will have fewer of these conflict spots than other tools. And once you get the command line quirks figured out, it's easy to script around them. But it's still awkward: effective, but awkward.

HTML comparison reports

The guiffy tree comparison feature is a very nice way to get a sense of what has changed in a project. Automatically generating an HTML document that looks like the GUI would be really nice. Putting the changes out where everyone can see them provides the kind of easy visibility that is important when project members aren't working tightly together.

guiffy's HTML reports don't look exactly like their GUI counterparts, but they sure are close:

The changes are highlighted using the same colors as the GUI, including the character-by-character highlighting. Obviously you can't change the highlight options in mid-stream, as you can with the GUI.

The directory comparison is a little less close to the GUI. It doesn't show the folder tree view. Instead, it looks more like the output of jiff -r. Regardless, it gets the information across:

The HTML reporting capability seems new, according to the release history, and I found some rough spots that support this:

Did you notice that the header says the folder compare report was produced by version 7.3—yet this is supposed to be a review of version 7.2? Well, I've got a beta! I found a bug in the HTML reporting: guiffy hangs when asked to generate folder reports in version 7.2. So I'm looking at an early image of the next version. (I didn't install the new .jar until time to come back and finish this section. Everything else for this review was generated on 7.2.)

I also found a problem with long lines. In the image below you can tell that a character has changed in the left file (because of the red color) but you can't see what it is:

This is a Firefox problem, though, not a Guiffy problem. The same page looks fine in Internet Explorer 6.0:

I had a look at the generated HTML, and it's clean. I think this one is a Firefox (actually, Gecko rendering engine) error.

Finally, errors like invalid filenames (or missing files, if you prefer) don't return a status code, they display a dialog box. This is definitely not good for a scripted solution.

This is a feature request, but I think it would make sense for the HTML report generator to produce a whole web of linked comparison reports: specify a switch, and get a whole series of output files, hyperlinked together, showing all the changed files.

The format of the generated HTML is very clean, and uses CSS tags for displaying changed entries. I think that a good perl coder could probably get the linkages going in a day or so. But it would be better if he didn't have to.

International Character Encodings

Consider this image of a guiffy compare of two versions of a file

What's interesting about this result is the nature of the inputs: one of these files is encoded as a plain old ASCII text file (technically, Windows codepage 1252) while the other is UTF16. Here's a different way of looking at them:

C:\\MyDocuments\\Sources\\Php Data Layer>sed -ne 41p field.uni.txt|od -t x1z
0000000 00 63 00 6c 00 61 00 73 00 73 00 20 00 46 00 69  >.c.l.a.s.s. .F.i<
0000020 00 65 00 6c 00 64 00 20 00 65 00 78 00 74 00 65  >.e.l.d. .e.x.t.e<
0000040 00 6e 00 64 00 73 00 20 00 4f 00 62 00 6a 00 65  >.n.d.s. .O.b.j.e<
0000060 00 63 00 74 00 20 00 69 00 6d 00 70 00 6c 00 65  >.c.t. .i.m.p.l.e<
0000100 00 6d 00 65 00 6e 00 74 00 73 00 20 00 50 00 65  >.m.e.n.t.s. .P.e<
0000120 00 72 00 73 00 69 00 73 00 74 00 61 00 62 00 6c  >.r.s.i.s.t.a.b.l<
0000140 00 65 00 20 00 7b 00 0d 00 0a                    >.e. .{....<
0000152

C:\\MyDocuments\\Sources\\Php Data Layer>sed -ne 41p field.php|od -t x1z
0000000 63 6c 61 73 73 20 46 69 65 6c 64 20 65 78 74 65  >class Field exte<
0000020 6e 64 73 20 42 61 73 65 20 69 6d 70 6c 65 6d 65  >nds Base impleme<
0000040 6e 74 73 20 50 65 72 73 69 73 74 61 62 6c 65 20  >nts Persistable <
0000060 7b 0a                                            >{.<
0000062

As you can see, the two files are totally different internally: two different character sets, two different encodings. By telling guiffy about the file encodings of each file—via the command line or a dialog box—the compare engine knows to ignore the encoding and focus on what's important: the contents of the files!

guiffy allows you to separately specify the encodings of the two primary files, the optional parent file for 3-way compare/merge operations, and the output file for merges. I recently worked at a client site where we would have been thankful just to be able to specify the input encodings: what a shame we didn't have guiffy.

One of the problems with multibyte characters is how to deal with “broken” or invalid character sequences. guiffy handles them fairly well: the invalid sequences are displayed as invalid characters. In this scenario you're probably going to need to hex dump the bytes to see exactly what went wrong (unless you just go fix the program that produced them).

The support for output encodings even extends to HTML reports. Specifying the output encoding produces an HTML document with the correct encoding and content-type metadata:

Or, you could look at it this way (notice the byte-order-mark: U+FEFF):

C:\\MyDocuments\\Sources\\Php Data Layer>head -1 diff.html | od -t x1z
0000000 fe ff 00 3c 00 21 00 44 00 4f 00 43 00 54 00 59  >...<.!.D.O.C.T.Y<
0000020 00 50 00 45 00 20 00 48 00 54 00 4d 00 4c 00 20  >.P.E. .H.T.M.L. <
0000040 00 50 00 55 00 42 00 4c 00 49 00 43 00 20 00 22  >.P.U.B.L.I.C. ."<
0000060 00 2d 00 2f 00 2f 00 57 00 33 00 43 00 2f 00 2f  >.-././.W.3.C././<
0000100 00 44 00 54 00 44 00 20 00 48 00 54 00 4d 00 4c  >.D.T.D. .H.T.M.L<
0000120 00 20 00 34 00 2e 00 30 00 31 00 20 00 54 00 72  >. .4...0.1. .T.r<
0000140 00 61 00 6e 00 73 00 69 00 74 00 69 00 6f 00 6e  >.a.n.s.i.t.i.o.n<
0000160 00 61 00 6c 00 2f 00 2f 00 45 00 4e 00 22 00 20  >.a.l././.E.N.". <
0000200 00 22 00 68 00 74 00 74 00 70 00 3a 00 2f 00 2f  >.".h.t.t.p.:././<
0000220 00 77 00 77 00 77 00 2e 00 77 00 33 00 2e 00 6f  >.w.w.w...w.3...o<
0000240 00 72 00 67 00 2f 00 54 00 52 00 2f 00 68 00 74  >.r.g./.T.R./.h.t<
0000260 00 6d 00 6c 00 34 00 2f 00 6c 00 6f 00 6f 00 73  >.m.l.4./.l.o.o.s<
0000300 00 65 00 2e 00 64 00 74 00 64 00 22 00 3e 00 0d  >.e...d.t.d.".>..<
0000320 00 0a

Looking at the release history of the product, the release 7.x series seems to have had a strong focus on support for international encodings. It sure looks like they did it right. On a side note, remember this illustration from above?

I cheated a little bit: those are actually Unicode “fullwidth” characters used when typesetting mixed Latin and CJK text. This has been an “internationalized” review from the very start!

Java API

I didn't explore the Java API for this review. I can document that it exists, and that it's very complete. It's called guiffy.inside, and it includes all the functions of all of the different flavors of the product, both GUI and command-line. It also includes a “hook” mechanism for modifying the behavior of the existing application.

On the command-line side, there are API equivalents for the jiff and suremerge commands so that you can create a windowless compare/merge subsystem inside your own programs. Also, there's a separate API for the HTML report generator.

There are two interfaces for GUI products. The first, called a Frame, essentially pops up the Guiffy application window, menus, dialog boxes, and all. The second, called a Panel, lets you put the Guiffy compare/merge screen inside your application's window. There's no menu, and no toolbars, in the Panel version: you get tight integration with your app, but you have to write the supporting code to pick up filenames and encodings and such.

Finally, there's a hook mechanism so that you can detect the merge result on exit and trigger some behavior. It seems Guiffy is just about as easy as can be to integrate with another application, like a CM tool or a build script or a continuous integration driver. Because this is all in Java, you can build it into an applet running on a web page. Make it a signed applet and your users can do compare and merge operations against their desktop, and you won't even have to install the product on their machines. (You should probably call and ask about licensing, though.)

Support

Guiffy support is just amazingly responsive!

I started this review by visiting the website (http://www.guiffy.com) and downloading the demo version, along with the sample scenarios for compare & merge. When I asked questions by e-mail, I was amazed to get responses at four o'clock in the morning. I was happy to find they were knowledgeable technical answers from someone who could understand what tasks I wanted to do, and how they should work. I also got responses in the afternoon and evening. If anyone in Missouri ever sleeps, you couldn't prove it by me.

Accuracy

SureMerge offers two different modes for grouping changes together. You can choose to see more blocks of changed text, with small numbers of lines, or you can permit grouping of changes into a smaller number of individually bigger blocks. I was unable to detect a difference in the modestly sized files I used for testing purposes.

For my purposes, the accuracy of the tool is 'pinpoint.' Changes are identified down to the individual characters that differ. It doesn't get any better than that.

Performance

SureMerge is a set of Java applications with a shell or Windows executable front-end. I was a little nervous about this, after having a bad time starting up Eclipse. SureMerge is definitely not Eclipse: it starts up just fine. Likewise, doing compares is quick even for large(-ish) files. I used some international dictionary files to test both large file operations and international character set support. It takes Guiffy about 15 seconds to re-compare two versions of a 1.7MB Russian dictionary on my two-year-old laptop. Very nice!

For this test, I wrote a perl script to randomly “mutate” about one percent of the lines: delete or move the line, or insert or transpose characters:

The help documentation includes notes for comparing large files when the files have large blocks of changes in them. The change is simply to let SureMerge allocate more memory to work in: everything else is taken care of. I guess that change is required if the files are more different than they are similar, because my files were way over the threshold for needing more memory, yet guiffy worked just fine out of the box.

Ease of Use

SureMerge is a tool with some complex features. Overall, the interface is very smooth and self-explanatory. Most advanced operations are explained well by the surrounding menus and dialog box text, with no need to refer to the online help.

That said, I have some complaints. (I'll bet you didn't see that coming.)

First is support for drag & drop. I'm using a modern JRE (1.5), and yet the drag & drop is all but unusable: the drop targets are the top and bottom row of pixels of the file name combo boxes on the file open dialog.

As I understand it from talking to the author, this is a defect in the Java UI libraries. That doesn't excuse it at the product level: a one-pixel drop target is not a drop target. As clunky as it may be, Guiffy needs to make this work—either code their own GUI widgets (blech!) or add an ugly “Drop Here” box (blech!).

Second, integrating guiffy with Windows Explorer is documented in a Tech Note in the on-line help. I believe the installer should perform the simple steps automatically on installation.

Finally, the dialog box for choosing character encodings needs work. Presently, the encodings are specified as text entry boxes: you type in the encodings. But since the list of supported encodings is known in advance, this should be a drop-down list or a combo list. It was frustrating to be told I needed an additional file:

It was even more frustrating to realize that the encoding I wanted was supported, just spelled differently ('koi8'). I'm sure that if you spend all day every day in a particular encoding you know how to spell it. But if you have to deal with different encodings on a regular basis, a pull-down list is your friend.

Overall, the product itself is very easy to use and install. But it could be better integrated with the desktop, and the user interface needs to be a little bit more helpful.

Effectiveness

SureMerge gets the job done. It gets the job done well. The tool is fast, it is accurate, and it is easy to use. Using the file-encoding feature lets you view changes at both the 'byte' level and at the 'code point' level. You can see that a character has changed, or you can switch views and see just exactly how the binary expression of that character has changed. For anyone dealing with i18n issues this is a godsend.

Included in the on-line help are instructions for integrating SureMerge with eight different CM tools. For PVCS, Guiffy even includes a specially built executable that replaces the pvcsmerge program for seamless integration. (Support for the other tools is probably just as seamless, but they offer configuration options to let you set the name of the merge tool so Guiffy doesn't have to ship eight different executables.)

The addition of batch reporting facilities and HTML report generation makes SureMerge a valuable tool for a whole team, not just individual developers. Now change sets and impact statements can be automatically displayed from an intranet dashboard. Continuous integration and nightly build scripts can be extended to produce clear, detailed, attractive code change summaries. Sweet!

Cost

Single-user licenses for SureMerge are USD $75. Price breaks occur at 3 users (3/$150) and again at 36 users. SureMerge is a good value for managers or team leads looking to provide a standard tool to their developers.

Conclusion

SureMerge is a solid, capable tool, as you would expect from a 7.2 release. It performs well and does a great job helping developers, build managers, team leads, and other players in the development cycle understand exactly what is going on with the code. The GUI itself is easy to use and is available in several languages. The tool is obviously growing, with international encoding support and HTML report generations being the most recent major features added. Guiffy, the company, is personified by owner Bill Ritcher: quick to respond, helpful, and willing to work with you to help get over any problems. I like the tool, and you will, too.

1) Yes, that's “goofy gooey.” I'm sorry, but it's Bill Ritcher's fault.

2) On the other hand, if you are a mad scientist trying to turn a monkey into an armadillo, this is the tool for you!

3) According to Bill Ritcher, this will be better documented in the 7.3 release.

[wiki | tools:byname:guiffy_suremerge]

Administrator — Fri, 18 Apr 2008 13:31:03 +0000

Guiffy SureMerge

SureMerge is a Java-based tool for both compare and merge operations. The tool is the primary focus of Guiffy Software. Austin Hastings wrote a review of the tool for the CM Crossroads web site. That review is available here, as well.

[wiki | tools:compare]

Administrator — Fri, 18 Apr 2008 12:50:09 +0000

Compare (diff) Tools

diff is of course the archetype of this class of tools. There are various implementations of the Unix command line diff utility, but the GNU diffutils package is considered representative of all of these. The tools are:

How do I review a SCM tool?

Austin Hastings — Thu, 27 Mar 2008 06:10:38 +0000

A while back, I wrote a review of Guiffy SureMerge for CM Crossroads. I even wrote about the process, here.

I’d like to think that I’m the guy for writing SCM tool reviews. I’ve been a vendor rep, so I know where a whole bunch of the bodies are buried. I am independent, so I don’t owe anybody anything. I have done the developer job, and the CM guy job, and the admin job, so I know what those guys care about. But I’m scared.

The fact is that there are a lot of details in any SCM tool. I spent two weeks reviewing SureMerge, which when you come right down to it is a glorified version of diff, for pete’s sake. Most SCM tools have some kind of diff utility built in as an afterthought. If it takes two weeks to do the afterthought, how many months would it take to cover the check-in command? When I wrote the case study that discussed the development of Longacre Deployment Management, I mentioned that the first-cut questionnaire we used for culling vendors contained about three hundred questions. Guess how many paragraphs that would translate to in a review.

One thing that seems obvious is that different vendors are at different points on the spectrum of object-version versus change-set based development. Some of them allow, or require, you to select which model you want. A bunch of them want to provide those functions, but call them something different and pretend that nobody else has them. Another thing is that the vendors, and even their customers, are at different levels with respect to integrating change tracking tools with version control. And with integrating requirements tools. And with integrating build tools. And deployment tools.

I’d like to have a fair measure. Some common set of activities that I could do for each tool. A benchmark, if you will, that allows the reader to hold two reviews up side-by-side and see the strengths of one tool versus another. (Ditto, weaknesses.) But there is the real risk that what challenges one tool will be a built in subfeature in a second level menu of a more advanced or more integrated tool. For instance, CVS doesn’t have any kind of change request tool integrated. There are some hacks. There are some other “products” that integrate CVS. (cvstrac, for one.) What to do? Accurev doesn’t have a CR tool, but they integrate with just about everything. MKS offers a separate CR tool, and makes it trivial to do change-set based development. Rational sells it as an add-on.

The point is, a “standard review” has the real risk of not mapping well to the product features. My SureMerge review was built around exploring the product, and deciding which features were worth mentioning to the readers. Some of them made it, others didn’t. A “custom review” runs the risk of spending a lot of time talking about worthless features because that is what is interesting about the particular tool. (I’ll mention here that a nice lady from PlasticSCM asked me to write a review of their 1.0 product. I declined for pretty much that reason—the product was too simple. Now they’re at version 2.0, and have a lot of new features. Surprise!)

Another point of concern is that some parts of reviewing a tool need other users involved. Access control, security, and privilege issues need multiple users. Too often, demo or review software has these features disabled or dumbed down. It’s hard to be in conflict with yourself. It’s hard to create any kind of log-jam with only one user. (And of course, it’s a considerable pain in the tuchus by yourself.)

So imagine a “standard” review of CVS: I try to create a set of change requests, and …

This sort of suggests that there maybe should be some kind of onion-peel review. The tool is what the tool is, and the reviewer peels away the standard layers until the tool starts to be relevant. But how much value does that even provide? If I’m writing a review of CVS, and the reader needs MKS Integrity or Telelogic Synergy, whose time is more wasted—mine or hers?

Bizarrely, then, the right first step seems to be to write a book (No, this isn’t a joke.) about SCM. Then match each tool against the book. But that’s crazy talk. The next best thing to writing a book, of course, is getting somebody else to write a book for me. There’s two ways to get there: pick on some standard text, or “web 2.0″ it. I can’t think of a useful “standard book,” so the web 2.0 approach doesn’t seem too bad. Perhaps the approach should be to create a wiki, fill it full of standard questions, and then encourage the users to answer them. Of course, most users aren’t very good writers, but isn’t that what wiki-gnomes are for? Of course, that puts me almost in competition with CM Crossroads, a position I don’t think I want to be in.

Maybe the right answer is a little of both. Create the wiki, then “edit up” when tasked to doing a particular review. Take the standard questions, and render them into useful text. Now I’m competing with Gartner. Joy!

If you’ve got some ideas, boy do I need ‘em.

MatchPoint – a first pass at deployment management?

Austin Hastings — Tue, 25 Mar 2008 15:10:27 +0000

I was browsing some Spanish-language software development blogs, and came across a reference to MatchPoint, a mainframe-oriented product from a Swiss company called CM First AG.

These guys don’t seem to have any more idea than most about how to manage database changes, but their product (or at least, the pdf of the glossy brochure) looks to have some really good tracking built in for deployments in both i-space and Windows-space. They claim that (for a small additional fee) you can track deployments all the way to the customer.

This is typical of a mainframe approach. The mainframe guys don’t have bajillions of PC customers, they have tens, or hundreds, or thousands of mainframe customers. Those are small enough numbers that it makes sense to talk about tracking them individually. It isn’t surprising, therefore, that mainframers have a more pragmatic approach to tracking deployments.

Thinking about how I could get my hands on a copy of MatchPoint (when I don’t own an IBM i-series) has led me to think some more about how to review a CM tool. I want to be able to write CM tool reviews, but I don’t want to write the kind of bunkum that you get from most 1500-word specials.

If you have any hands-on experience with MatchPoint, I’d love to hear from you — to get some real world feedback on their glossy brochure, if nothing else. And if you have ideas on how to review a CM tool, well, I’ll write something up in a separate post. Hold that thought.