How do I review a SCM tool?
A while back, I wrote a review of Guiffy SureMerge for CM Crossroads. I even wrote about the process, here.
I’d like to think that I’m the guy for writing SCM tool reviews. I’ve been a vendor rep, so I know where a whole bunch of the bodies are buried. I am independent, so I don’t owe anybody anything. I have done the developer job, and the CM guy job, and the admin job, so I know what those guys care about. But I’m scared.
The fact is that there are a lot of details in any SCM tool. I spent two weeks reviewing SureMerge, which when you come right down to it is a glorified version of diff, for pete’s sake. Most SCM tools have some kind of diff utility built in as an afterthought. If it takes two weeks to do the afterthought, how many months would it take to cover the check-in command? When I wrote the case study that discussed the development of Longacre Deployment Management, I mentioned that the first-cut questionnaire we used for culling vendors contained about three hundred questions. Guess how many paragraphs that would translate to in a review.
One thing that seems obvious is that different vendors are at different points on the spectrum of object-version versus change-set based development. Some of them allow, or require, you to select which model you want. A bunch of them want to provide those functions, but call them something different and pretend that nobody else has them. Another thing is that the vendors, and even their customers, are at different levels with respect to integrating change tracking tools with version control. And with integrating requirements tools. And with integrating build tools. And deployment tools.
I’d like to have a fair measure. Some common set of activities that I could do for each tool. A benchmark, if you will, that allows the reader to hold two reviews up side-by-side and see the strengths of one tool versus another. (Ditto, weaknesses.) But there is the real risk that what challenges one tool will be a built in subfeature in a second level menu of a more advanced or more integrated tool. For instance, CVS doesn’t have any kind of change request tool integrated. There are some hacks. There are some other “products” that integrate CVS. (cvstrac, for one.) What to do? Accurev doesn’t have a CR tool, but they integrate with just about everything. MKS offers a separate CR tool, and makes it trivial to do change-set based development. Rational sells it as an add-on.
The point is, a “standard review” has the real risk of not mapping well to the product features. My SureMerge review was built around exploring the product, and deciding which features were worth mentioning to the readers. Some of them made it, others didn’t. A “custom review” runs the risk of spending a lot of time talking about worthless features because that is what is interesting about the particular tool. (I’ll mention here that a nice lady from PlasticSCM asked me to write a review of their 1.0 product. I declined for pretty much that reason—the product was too simple. Now they’re at version 2.0, and have a lot of new features. Surprise!)
Another point of concern is that some parts of reviewing a tool need other users involved. Access control, security, and privilege issues need multiple users. Too often, demo or review software has these features disabled or dumbed down. It’s hard to be in conflict with yourself. It’s hard to create any kind of log-jam with only one user. (And of course, it’s a considerable pain in the tuchus by yourself.)
So imagine a “standard” review of CVS: I try to create a set of change requests, and …
This sort of suggests that there maybe should be some kind of onion-peel review. The tool is what the tool is, and the reviewer peels away the standard layers until the tool starts to be relevant. But how much value does that even provide? If I’m writing a review of CVS, and the reader needs MKS Integrity or Telelogic Synergy, whose time is more wasted—mine or hers?
Bizarrely, then, the right first step seems to be to write a book (No, this isn’t a joke.) about SCM. Then match each tool against the book. But that’s crazy talk. The next best thing to writing a book, of course, is getting somebody else to write a book for me. There’s two ways to get there: pick on some standard text, or “web 2.0″ it. I can’t think of a useful “standard book,” so the web 2.0 approach doesn’t seem too bad. Perhaps the approach should be to create a wiki, fill it full of standard questions, and then encourage the users to answer them. Of course, most users aren’t very good writers, but isn’t that what wiki-gnomes are for? Of course, that puts me almost in competition with CM Crossroads, a position I don’t think I want to be in. 
Maybe the right answer is a little of both. Create the wiki, then “edit up” when tasked to doing a particular review. Take the standard questions, and render them into useful text. Now I’m competing with Gartner. Joy!
If you’ve got some ideas, boy do I need ‘em.