So here’s which.cmd – a script that tries to DWIW.

Note that this script is wrong in one key detail: it searches inside-out. That is, any “foo.COM” will take precedence over any “foo.EXE” even if the .exe version occurs earlier in the PATH.

@echo off
REM Copyright (c) 2010, Austin Hastings.
REM This file may be used for any purpose without restriction.

REM NOTE: This script does not handle multiple entries with different
REM extensions correctly. It returns the first matching EXTENSION, rather
REM than returning the first DIRECTORY with any EXTENSION.

REM Analogous to unix 'which' command, look for a matching runnable
REM in %PATH% and print the location.

SETLOCAL ENABLEDELAYEDEXPANSION

set check_extensions= ;%PATHEXT%

:ext_loop
if "%check_extensions%" == "" goto done

for /F "tokens=1* delims=;" %%E in ( "%check_extensions%" ) do (
	set check_extensions=%%F
	set target=%1%%E
	for %%W in (  !target! ) do set answer=%%~f$PATH:W
)

if "%answer%" == "" goto ext_loop

:done
if "%answer%" == "" goto not_found

@echo %answer%
goto exit

:not_found
@echo No runnable matching '%1' was found.

:exit

Brad Appleton is, in my opinion, one of the all-around smart guys in the CM space. His focus for the last few years has been on Agile CM, but his writings are applicable to anybody doing software development, for a very broad definition of software – most of what he writes is applicable to almost any kind of intellectual property. And recently Brad has been doing some blogging about books on various flavors of ‘trust. ‘

Trust, to me, is one of those core values for most CM specialists. The fact is that CM is a simple job. There are certain requirements, and once you meet them you get to go home. In that way it’s a lot like being a system administrator: is the system up, is everything working? Okay, go home. Now admittedly it can be pretty hard to meet some of those requirements. That’s why I didn’t say it was an easy job – just a simple one. But that’s where trust comes in. Because whether your shop is an agile shop or not, the CM guys are more affected by trust than any single other thing.

Let me repeat that: CM guys are more affected by trust than any single other thing.

I’m a CM consultant. I get called in by shops that either don’t have CM and want to build a team, or by shops that already HAVE a CM team but have experienced enough CM failures that they don’t trust them any more. Since most companies obtain headcount by calling up a body shop and ordering it by the pound (kilos for the EU, I guess), I don’t usually get called first for CM team setups. It’s usually only after there are some bodies on site, when it turns out that CM specialists that are great at keeping an existing system going don’t have so much experience in starting a new system up from scratch. So there’s usually some kind of mess on the floor when I show up.

What does that mean for you? Well, it means that if you see me at your shop, you should make sure your resume is up to date. (Sorry, but it’s true.) It also means that you can keep me, or somebody like me, from showing up if you concentrate on building trust with your customers.

Brad’s first blog post about trust was on Covey’s “The Speed of Trust,” and in that book there are five “rings of trust” outlined. The second and third rings, or types, are interpersonal trust and organizational trust. That’s where I live, and if you’re smart it’s where you spend time each day maintaining your relationships with your customers.

It’s a pretty simple set of questions, but it makes all the difference.

1. Do the people you interact with understand what you need from them, and understand how to deliver it?
2. Do they understand what services you can provide to them, and how to request them?
3. Are all of these things practicable?
4. Do they perceive you as doing what is needed, on time, and without unnecessary oversight?
5. Do they perceive you as requesting things that are objective, and consistent, and in keeping with your and their function?
6. Do they believe the estimates and the explanations you give them?

If you can answer yes to all of those, you’ll never work with me. That’s because your customers trust you. If you can’t – if your customers distrust you – then as the level of trust decreases, the likelihood of a “fixer” being called in for your team goes up. In the credit card business, your interest rate goes up. In the CM business, the level of interest in working with you goes down.

If you’re a CM manager for a military systems contractor, then you’ve got a CM plan. And you’ve got a set of prescribed deliverables. And if the developers – software, firmware, systems engineers, whatever – know how to deliver each release to you, and it’s an automated, mechanical, simple chore, then you’re halfway there. If your other customers – the clients, and the development team that need to roll back to an earlier release – can submit a request and quickly get some kind of link, or bundle, or set of typing instructions, then you’re all the way done.

If you’re a build guy, or a release guy, or a deployment guy for a software team – agile or not – then the activities might be different but the questions aren’t. Does the team know how to get you to do a build, or cut a release, or deploy a package? Are there any bogus hoops they have to jump through? Have you built any passive-aggressive walls, saying “we want the development team to do this thing which they really ought to trust us to do but they overrode us that one time and now we’re bitter and whiny and nobody loves us”? Can a developer quickly get to the codeface of some old branch from six months ago?

I guess the short form is this. Are you doing your job? Are you making it as easy as possible for your customers to ask you to do your job for them? Because in my experience – and I’ll admit my experiences are all negative – if you aren’t doing your job, or if you’re making your customers pay some kind of “tax” (time, whining, paperwork, hoop-jumping, sloth, inefficiency) then you’re going to lose trust. And once you lose trust, you’ll see a little bureaucracy get started. And when that doesn’t work (and it rarely does) you’ll get to meet me. I’ll be the guy “we brought in to help gets things caught up.”

This isn’t a threat, quite. Because I don’t know you. It’s a reflection of the world as I see it, because I’m like a specialist doctor – nobody makes an appointment with a gastroenterologist to talk about how healthy they are. In my case, nobody calls me up and says, “We’d like to pay you to fly out here and sit in a conference room and live in a hotel for two weeks so that we can tell you how smoothly our build and deployment works!” (Man, I would LOVE that.) But what I get instead is, “We need you to come and fix our build and deployment procedures. Our developers don’t trust the CM team, and we’ve had a couple of blown deployments in the last few months.”

So anyway, I guess the upshot of this blog post – and this is much too depressing to be a CM Journal article, so it’ll stay a blog post – is that you need to make sure your house is in order, trust-wise. Go read one or more of the books on Brad’s list. Because CM teams seem far more Boolean than development teams. Developers can say “It’ll be done soon,” while CM’ers have to say “it’s not done.” And saying “it’s not done” leads pretty directly to “you’re not hired.”

My client was reorganizing a development team into an agile mode in order to deal with an outsourcing disaster. It was one of those horror stories that starts out like “We were 3 years in to a 10 month project, when …” I was brought on to help ramp up their build and deployment processes, to get continuous integration and testing systems up, and to help mechanize their deployments.

The impetus for the MySQL unit test framework came from a short term contractor. The one “constant” in the otherwise highly-variable environment had been that the team wasn’t changing. Suddenly, that stopped being true. We had a handful of contractors in for a short term surge, and none of them were familiar with the application, or with SQL (it seemed). The offender was using the CI server as his syntax checker for database scripts. Yikes!

Luckily, the garbage he delivered was syntactically invalid. The mysql command-line client rejected the script and exited with an error status, which caused the CI script to log a failure, and … voila! Everyone credited me with doing a great job of implementing CI, because it was even checking the database stuff. Like a good consultant, I said “Yes. Of course.” Then I ran back to my desk, thinking “Oh, no! I’ve got to come up with a unit test framework for the MySQL stuff, pronto!” And I did.

Solutions

The software was a pretty complex web app, and the target environment was a mixture of Linux and Windows. As a result, when I looked at existing SQL unit test frameworks (dbUnit and sqlUnit, respectively) I had to reject them. They are decent products – and free – but the fact is that they expect certain things about their runtime environment that we couldn’t deliver. We had to build our own MySQL unit test framework, ideally in “pure” MySQL.

Fortunately for me, a nice guy named Giuseppe Maxia maintains a web site at datacharmer.org, and he has written a library of general-purpose MySQL code. (Called the General Purpose Stored Routine library, or gp_sr_lib.) Among the all this well-written, free, already debugged code was a set of testing routines. Thirty minutes into my development effort, I was already 80% done. Woo-hoo!

The GP library test routines are a good start on a mysqlUnit library, and I strongly recommend that if you need such a library you stop reading here and go download the code. Right now.

Rolling (y)our Own

There are a couple of gotchas that you’ll want to watch out for in your own unit testing library. First, the tests should be able to integrate into your CI engine. In particular, you need to decide on and implement some kind of failure mode – either fail when the first problem occurs, or fail after running all test cases. If you’re running the mysql client via some kind of exec call, be aware that while syntax errors will cause the client to exit with a failure code, other errors will not cause a failure. Instead, the client prints a diagnostic and returns zero. What’s more, the syntax does not support a deliberate abort from within the SQL engine.

In my own case, I chose to follow the model that Maxia has provided, and NOT fail as soon as a test fails. Rather, my code collected the results of all the tests that were run, and summarized them. In order to get “fail on error” behavior out of the mysql command line client, I coded a routine that would perform a “SELECT force_mysql_client_abort FROM no_such_table”. This definitely does cause the client to abort, at least until some developer creates a table called “no_such_table.” Then I wrapped the test cases in an “ignore failure” script that called the summarizer after all the cases had run, which in turn would call the abort routine if the error count was non-zero.

Another thing I chose to “fix” was the routine names. Maxia’s code includes both functions and procedures, but the names are not really in keeping with the style used in the various xUnit test frameworks. The library contains a procedure for asserting the existence of a database table. It is called as:

CALL check_table('db name', 'table name');


I opted to “embrace and extend” the library. I wrapped these types of checks into functions, like this:


set_database('db name');
assert_table_exists('table name');
assert_table_exists_in_db('other table', 'other db');


And in general, wherever there is a straight-ahead assert, I also added an “assert_not”. In terms of effect, there is no real difference in behavior between Maxia’s naming and my own. But I felt that the developers, who were using jUnit and nUnit, would be more comfortable learning a set of routines that had a similar structure and style.

Going Most of the Way: DDL

I tried to make a list of the various things that SQL scripts might do. The most obvious — and this probably accounted for two thirds or more of the scripts we had — is DDL statements. DDL (Data Definition Language) is that subset of SQL that relates to defining databases, tables, indexes, triggers, and the like. While the most frequently executed SQL statements are DML (Data Manipulation Language), DDL is where it’s at for most of the human-generated script files: CREATE TABLE, ADD INDEX, etc.

The great thing is that the assertions for these statements are trivial to write. Maxia’s library includes the CHECK_TABLE procedure, and some code for looking up procedures and functions. Adding other stuff, like keys and constraints, is pretty straightforward. By the end of the day you should have a long list of functions like assert_table_exists, assert_index_exists, assert_database_exists, and assert_column_has_type.

Deploy Early

At this point, you’re ready to go live. Except that you need to build a framework to run these tests. That’s going to depend a lot on your deployment process, and on your particular CI environment. Sorry. The simplest idea would be to write a script that matches the file names of test files, and runs them one at a time. Call that script from CI, and make sure the exit status is meaningful.

Forget the next couple of sections, because most of the database changes you’re likely to see are DDL. Developers and DBAs do a lot of adding fields, adding indexes, and sometimes adding tables. If you have just the existence assertions for each kind of object, your library will cover two-thirds or more of the statements being delivered. That’s not a trivial amount of test coverage for a day or two of work. Ship it! Your developers will need some training, and you’ll need to beat on the thing once it’s deployed to your CI servers to make sure that the errors are coming out correctly.

Sell to the Development Team

I recommend that you write some simple tests of things you know to be true. Confirm the positive cases first. Then sit down with a smart, involved developer and pair develop some unit tests for code he hasn’t written yet. Do it the way you want the developers to do it, including deciding how to handle assertions that don’t exist yet.

In our case, the DBAs had already established a naming convention for database update files. I chose to duplicate the file names, which would look like ####_description.sql, changing the .sql extension to .test.sql. This let the testing framework associate the test with the delivered change. When running a “from zero” deployment, the test framework would then run TEST-CHANGE-TEST and expect the first test to fail and the second to pass. This demonstrated that the test was meaningful for the change (remember, the change is supposed to repair a shortcoming that is demonstrated by the test).

Once you’ve got buy-in from your chosen developer, sit down and develop a presentation that the two of you can give. Point to specific examples of blown deployments as reasons for unit testing the SQL. Then get the coder to explain the available test assertions. Then explain the framework you’ve developed to integrate the testing library with CI, and explain what failures are going to look like. And don’t forget to point them at the documentation you’ve written on the project wiki. This should get your team on board, and now you can start looking for other assertions to write. You can also start tightening the screws on your framework. Make it so that no change can be delivered without at least one test.

Simple DML Checking

The next obvious category of SQL statements to check is simple DML. These are the CRUD verbs: INSERT, SELECT, UPDATE and DELETE. Except that SELECT can be pretty hard to check within SQL, so it’s best to ignore it for now in favor of the others. The problem with DML is knowing what to check for. If you insert a row, should you check that the ROW_COUNT() returns 1? Or should you query for the particular key of that row?

I’m pretty convinced that DML commands in scripts are going to be metadata related. That is, your application may be inserting customer records, or video titles, or whatever you track. But if a developer codes a DELETE, INSERT, or UPDATE into a deployable script, I think the intent is either to perform some kind of bulk fix (convert data fields, or eliminate nulls) or it is adjusting the “background” data — the list of postal codes, state and province names, etc. that your application relies on. In the case of the simple background data, if you added or updated it, you should check it explicitly: assert that for a specific key, the values are as expected.

Some examples of helpful functions would be assert_statement_affects_rowcount(prepared_statement, num_rows), assert_table_column_matches_count('table name', 'column', value, count), and their negatives. The way to do dynamic programming in MySQL is with prepared statements. You can construct a string containing a SQL statement, use PREPARE to assign it to an identifier, and then EXECUTE the identifier:


PREPARE stmt
FROM
CONCAT("SELECT COUNT(*) FROM ", table_name, " WHERE ", field_name, " = ?");



EXECUTE stmt USING value;


In the more complex case of a data format or type update, the update should be encoded in a stored procedure. This way you can unit test the stored procedure with sample data. Once the procedure is ready to go, write a script that calls the procedure – confident already that the procedure will work – and test the invocation by confirming random elements of the data have converted correctly.

Testing Result Sets

As mentioned, testing SELECT can be a real challenge. While it is easy to create an assert_row_count(num) to confirm that your query returned the right number of values, it can be much harder to create a function to assert that a particular ordering exists, or that the results are truly grouped by postal code.

The first thing to do is go ahead and write the assertion functions for number of rows returned by the query. That’s simple, straightforward, and it can stand in place of more complex stuff for a long time. You may not ever get around to writing the more challenging assertions — if the tests are designed well enough, the number of results may be meaningful enough. If you do need to write more, especially if the query is complex, pass the query as a string to your assertion, prepare it as a statement, and make sure the results go into a temporary table. Then you can write whatever horribly complex code you need against the temporary table, without having to worry about losing context.

Testing Triggers

Triggers are going to be harder to test than other pieces of code because of the diverse ways you can invoke them. For example, a DELETE trigger may need to be checked with a simple delete statement as well as with a CASCADE from a foreign key relationship. Beyond creating assertions for the existence and type of triggers (BEFORE/AFTER, INSERT/UPDATE/DELETE, etc.) there isn’t really any obvious set of assertions to write. It’s more a question of deciding how to test the effects of the triggers, and of documenting the ways the trigger can be invoked.

Conclusion

Unit testing, and test-first development, are well established best practices. But it’s easy to overlook the “small stuff” when developers are setting up frameworks for testing their code. Database updates are small, but when they go wrong they can take out the rest of the system. It’s important to apply the same techniques, and get the same value, to this part of your development. Now you can.

First, there’s the Telelogic User Group Conference 2008, in Austin, Texas. That’s obviously focused on Telelogic, and on CM Synergy and Telelogic Change, or whatever it’s called this week. Since I spent a fair amount of time during the initial implementation of LDM wishing that I could use the kind of one-off ad hoc queries that Change is so good at, I suppose it’s appropriate.

The second conference is CMconf.DE, in Munich, Germany. Sadly, it will be a few weeks after Oktoberfest — otherwise there would be no hope at all of getting hotel space, I suppose. I’ll just have to wander around looking for any leftover barrels of beer, slightly-used lederhosen, or oompa bands that I can scrounge up. The theme is the same — presenting LDM and some other approaches to database CM.

In both cases, the time constraints are really biting. LDM needs an understanding of the other ways to do (fail at) database CM before I can really talk about it, and I’m pretty sure that most people haven’t thought beyond one way, if that many. So I’ll be speaking really, really fast, and inviting attendees to stick around afterwards to talk about it.

Anyway, that’s two continents in a single month — not bad, if I do say so. If I could just find a conference in India, China, or Israel, I could sweep the northern hemisphere.