Unit tests for MySQL scripts

Austin Hastings — Sun, 26 Apr 2009 01:07:09 +0000

Recently I had the opportunity to develop a unit testing framework for MySQL scripts in mostly-pure SQL. While I can’t share the code, I can certainly describe what we did, and why we did it. Hopefully it will be useful to you.

My client was reorganizing a development team into an agile mode in order to deal with an outsourcing disaster. It was one of those horror stories that starts out like “We were 3 years in to a 10 month project, when …” I was brought on to help ramp up their build and deployment processes, to get continuous integration and testing systems up, and to help mechanize their deployments.

The impetus for the MySQL unit test framework came from a short term contractor. The one “constant” in the otherwise highly-variable environment had been that the team wasn’t changing. Suddenly, that stopped being true. We had a handful of contractors in for a short term surge, and none of them were familiar with the application, or with SQL (it seemed). The offender was using the CI server as his syntax checker for database scripts. Yikes!

Luckily, the garbage he delivered was syntactically invalid. The mysql command-line client rejected the script and exited with an error status, which caused the CI script to log a failure, and … voila! Everyone credited me with doing a great job of implementing CI, because it was even checking the database stuff. Like a good consultant, I said “Yes. Of course.” Then I ran back to my desk, thinking “Oh, no! I’ve got to come up with a unit test framework for the MySQL stuff, pronto!” And I did.

Solutions

The software was a pretty complex web app, and the target environment was a mixture of Linux and Windows. As a result, when I looked at existing SQL unit test frameworks (dbUnit and sqlUnit, respectively) I had to reject them. They are decent products – and free – but the fact is that they expect certain things about their runtime environment that we couldn’t deliver. We had to build our own MySQL unit test framework, ideally in “pure” MySQL.

Fortunately for me, a nice guy named Giuseppe Maxia maintains a web site at datacharmer.org, and he has written a library of general-purpose MySQL code. (Called the General Purpose Stored Routine library, or gp_sr_lib.) Among the all this well-written, free, already debugged code was a set of testing routines. Thirty minutes into my development effort, I was already 80% done. Woo-hoo!

The GP library test routines are a good start on a mysqlUnit library, and I strongly recommend that if you need such a library you stop reading here and go download the code. Right now.

Rolling (y)our Own

There are a couple of gotchas that you’ll want to watch out for in your own unit testing library. First, the tests should be able to integrate into your CI engine. In particular, you need to decide on and implement some kind of failure mode – either fail when the first problem occurs, or fail after running all test cases. If you’re running the mysql client via some kind of exec call, be aware that while syntax errors will cause the client to exit with a failure code, other errors will not cause a failure. Instead, the client prints a diagnostic and returns zero. What’s more, the syntax does not support a deliberate abort from within the SQL engine.

In my own case, I chose to follow the model that Maxia has provided, and NOT fail as soon as a test fails. Rather, my code collected the results of all the tests that were run, and summarized them. In order to get “fail on error” behavior out of the mysql command line client, I coded a routine that would perform a “SELECT force_mysql_client_abort FROM no_such_table”. This definitely does cause the client to abort, at least until some developer creates a table called “no_such_table.” Then I wrapped the test cases in an “ignore failure” script that called the summarizer after all the cases had run, which in turn would call the abort routine if the error count was non-zero.

Another thing I chose to “fix” was the routine names. Maxia’s code includes both functions and procedures, but the names are not really in keeping with the style used in the various xUnit test frameworks. The library contains a procedure for asserting the existence of a database table. It is called as:

CALL check_table('db name', 'table name');

I opted to “embrace and extend” the library. I wrapped these types of checks into functions, like this:

set_database('db name'); assert_table_exists('table name'); assert_table_exists_in_db('other table', 'other db');

And in general, wherever there is a straight-ahead assert, I also added an “assert_not”. In terms of effect, there is no real difference in behavior between Maxia’s naming and my own. But I felt that the developers, who were using jUnit and nUnit, would be more comfortable learning a set of routines that had a similar structure and style.

Going Most of the Way: DDL

I tried to make a list of the various things that SQL scripts might do. The most obvious — and this probably accounted for two thirds or more of the scripts we had — is DDL statements. DDL (Data Definition Language) is that subset of SQL that relates to defining databases, tables, indexes, triggers, and the like. While the most frequently executed SQL statements are DML (Data Manipulation Language), DDL is where it’s at for most of the human-generated script files: CREATE TABLE, ADD INDEX, etc.

The great thing is that the assertions for these statements are trivial to write. Maxia’s library includes the CHECK_TABLE procedure, and some code for looking up procedures and functions. Adding other stuff, like keys and constraints, is pretty straightforward. By the end of the day you should have a long list of functions like assert_table_exists, assert_index_exists, assert_database_exists, and assert_column_has_type.

Deploy Early

At this point, you’re ready to go live. Except that you need to build a framework to run these tests. That’s going to depend a lot on your deployment process, and on your particular CI environment. Sorry. The simplest idea would be to write a script that matches the file names of test files, and runs them one at a time. Call that script from CI, and make sure the exit status is meaningful.

Forget the next couple of sections, because most of the database changes you’re likely to see are DDL. Developers and DBAs do a lot of adding fields, adding indexes, and sometimes adding tables. If you have just the existence assertions for each kind of object, your library will cover two-thirds or more of the statements being delivered. That’s not a trivial amount of test coverage for a day or two of work. Ship it! Your developers will need some training, and you’ll need to beat on the thing once it’s deployed to your CI servers to make sure that the errors are coming out correctly.

Sell to the Development Team

I recommend that you write some simple tests of things you know to be true. Confirm the positive cases first. Then sit down with a smart, involved developer and pair develop some unit tests for code he hasn’t written yet. Do it the way you want the developers to do it, including deciding how to handle assertions that don’t exist yet.

In our case, the DBAs had already established a naming convention for database update files. I chose to duplicate the file names, which would look like ####_description.sql, changing the .sql extension to .test.sql. This let the testing framework associate the test with the delivered change. When running a “from zero” deployment, the test framework would then run TEST-CHANGE-TEST and expect the first test to fail and the second to pass. This demonstrated that the test was meaningful for the change (remember, the change is supposed to repair a shortcoming that is demonstrated by the test).

Once you’ve got buy-in from your chosen developer, sit down and develop a presentation that the two of you can give. Point to specific examples of blown deployments as reasons for unit testing the SQL. Then get the coder to explain the available test assertions. Then explain the framework you’ve developed to integrate the testing library with CI, and explain what failures are going to look like. And don’t forget to point them at the documentation you’ve written on the project wiki. This should get your team on board, and now you can start looking for other assertions to write. You can also start tightening the screws on your framework. Make it so that no change can be delivered without at least one test.

Simple DML Checking

The next obvious category of SQL statements to check is simple DML. These are the CRUD verbs: INSERT, SELECT, UPDATE and DELETE. Except that SELECT can be pretty hard to check within SQL, so it’s best to ignore it for now in favor of the others. The problem with DML is knowing what to check for. If you insert a row, should you check that the ROW_COUNT() returns 1? Or should you query for the particular key of that row?

I’m pretty convinced that DML commands in scripts are going to be metadata related. That is, your application may be inserting customer records, or video titles, or whatever you track. But if a developer codes a DELETE, INSERT, or UPDATE into a deployable script, I think the intent is either to perform some kind of bulk fix (convert data fields, or eliminate nulls) or it is adjusting the “background” data — the list of postal codes, state and province names, etc. that your application relies on. In the case of the simple background data, if you added or updated it, you should check it explicitly: assert that for a specific key, the values are as expected.

Some examples of helpful functions would be assert_statement_affects_rowcount(prepared_statement, num_rows), assert_table_column_matches_count('table name', 'column', value, count), and their negatives. The way to do dynamic programming in MySQL is with prepared statements. You can construct a string containing a SQL statement, use PREPARE to assign it to an identifier, and then EXECUTE the identifier:

PREPARE stmt FROM CONCAT("SELECT COUNT(*) FROM ", table_name, " WHERE ", field_name, " = ?");

EXECUTE stmt USING value;

In the more complex case of a data format or type update, the update should be encoded in a stored procedure. This way you can unit test the stored procedure with sample data. Once the procedure is ready to go, write a script that calls the procedure – confident already that the procedure will work – and test the invocation by confirming random elements of the data have converted correctly.

Testing Result Sets

As mentioned, testing SELECT can be a real challenge. While it is easy to create an assert_row_count(num) to confirm that your query returned the right number of values, it can be much harder to create a function to assert that a particular ordering exists, or that the results are truly grouped by postal code.

The first thing to do is go ahead and write the assertion functions for number of rows returned by the query. That’s simple, straightforward, and it can stand in place of more complex stuff for a long time. You may not ever get around to writing the more challenging assertions — if the tests are designed well enough, the number of results may be meaningful enough. If you do need to write more, especially if the query is complex, pass the query as a string to your assertion, prepare it as a statement, and make sure the results go into a temporary table. Then you can write whatever horribly complex code you need against the temporary table, without having to worry about losing context.

Testing Triggers

Triggers are going to be harder to test than other pieces of code because of the diverse ways you can invoke them. For example, a DELETE trigger may need to be checked with a simple delete statement as well as with a CASCADE from a foreign key relationship. Beyond creating assertions for the existence and type of triggers (BEFORE/AFTER, INSERT/UPDATE/DELETE, etc.) there isn’t really any obvious set of assertions to write. It’s more a question of deciding how to test the effects of the triggers, and of documenting the ways the trigger can be invoked.

Conclusion

Unit testing, and test-first development, are well established best practices. But it’s easy to overlook the “small stuff” when developers are setting up frameworks for testing their code. Database updates are small, but when they go wrong they can take out the rest of the system. It’s important to apply the same techniques, and get the same value, to this part of your development. Now you can.

Two Conferences

Austin Hastings — Wed, 17 Sep 2008 02:15:52 +0000

Austin Hastings (that’s me) will be speaking at two conferences coming up in October. Both presentations will be focusing on LDM.

First, there’s the Telelogic User Group Conference 2008, in Austin, Texas. That’s obviously focused on Telelogic, and on CM Synergy and Telelogic Change, or whatever it’s called this week. Since I spent a fair amount of time during the initial implementation of LDM wishing that I could use the kind of one-off ad hoc queries that Change is so good at, I suppose it’s appropriate.

The second conference is CMconf.DE, in Munich, Germany. Sadly, it will be a few weeks after Oktoberfest — otherwise there would be no hope at all of getting hotel space, I suppose. I’ll just have to wander around looking for any leftover barrels of beer, slightly-used lederhosen, or oompa bands that I can scrounge up. The theme is the same — presenting LDM and some other approaches to database CM.

In both cases, the time constraints are really biting. LDM needs an understanding of the other ways to do (fail at) database CM before I can really talk about it, and I’m pretty sure that most people haven’t thought beyond one way, if that many. So I’ll be speaking really, really fast, and inviting attendees to stick around afterwards to talk about it.

Anyway, that’s two continents in a single month — not bad, if I do say so. If I could just find a conference in India, China, or Israel, I could sweep the northern hemisphere.

Doing better » database

Unit tests for MySQL scripts

Two Conferences

Microsoft's take on database change management