Doing better

Some bash goodness

Austin Hastings — Wed, 28 Oct 2009 15:09:48 +0000

Here’s some bash goodness (well, not really) to make ‘less’ a little bit more useful.

less() {

local -a args

for arg

case “$arg” in

*:[[:digit:]]* )

line=${arg/#+(?):/}

args[${#args[*]}]=”+$line”

arg=${arg/%:+(?)/}

;;

esac

args[${#args[*]}]=$arg

done

$( which less ) “${args[@]}”

}

Basically, this shell function loads when you tell it to (in your .profile, likely), and it replaces the ‘less’ command. When you type ‘less …’ on the command line, the function runs.

It scans through the args, looking for one like filename.c:24, and if it finds that kind of arg, it replaces it with +24 filename.c — translating the syntax used by a *lot* of compilers into the syntax used by less for opening a file and jumping directly to the line number.

Why not 2 stacks?

Austin Hastings — Wed, 01 Jul 2009 00:05:13 +0000

I was googling around the other day for compiler wisdom when I stumbled on a computer security “research” paper. Some one or ones were postulating some theoretical result about vulnerability to buffer overflow attacks in blah, blah, blah special case. I didn’t read it, because that’s not what I was after.

But it made me think: how come Microsoft hasn’t just solved this problem already? Two “stacks” would wipe out the overflow attack problem at minimal cost.

Back Story

If you don’t know, here’s a very (!) short summary of the problem.

Computer software uses “subroutines” to do everything. In fact, even the “main” program is a subroutine. Because subroutines can call other subroutines, they can “stack up”. That is, “main” can call “sell ticket” which can call “print schedule” which can call “draw table”, etc.

What’s a stack?

In modern systems, there is no really good way to keep track of how many routines will get stacked up, a general-purpose data structure called a “stack” is used. This is so important that CPU’s explicitly support a single “call stack” with special CPU instructions like push and pop, special registers to access the stack, and all kinds of incredibly expensive stuff.

A stack works by “stacking” data, in much the same way they stack plates at a buffet restaurant (or cafeteria). The only data item you can get to is the one at the “top” of the stack. (The only plate you can take is the one on the top of the stack. See?) If you want to put on more data, it goes on the top of the stack, and the stack grows. If you want to take off data, you take it off the top of the stack, and the stack shrinks.

In this case, though, there are two ways to think about what “data” means. One way is to think of data as being individual units. One number goes on the stack, then another number goes on top of it, etc. But another way is to think of data “sets” going on the stack. If you think about the back button on your web browser, it acts like a stack. You click on a link, and a page gets pushed on to the “history” in the browser. You click “Back” and a page gets popped off the stack, leaving a different page visible in the browser. But these pages aren’t just a single number. There’s a URL, the page contents, maybe some form data, etc.

In terms of computer subroutines, each subroutine may have some “local data” that it needs to store. The URL of a web page, or the name of the current user, for example. And this data is expected to be used within the subroutine, maybe passed down into any sub-subroutines that get called, and then it will disappear when the subroutine finishes.

So a subroutine may have a bunch of local data, much like a web page in your browers. That can also be managed as part of a stack, using what is called a stack frame. A stack frame is a collection of data that all gets jammed onto the stack at one time, so the subroutine can use it. Again,”modern CPUs” (pretty much all of them built since the 1970’s) support this directly.

How important is the stack?

The de facto standard computer in the world today is the Intel x86 CPU of some kind: Pentium, Opteron, whatever you want to call it. Back when dinosaurs roamed the earth and nobody thought a 16-bit computer was worth the extra money over a 8-bit computer, the x86 family had about 30,000 transistors on board. Now, when people are arguing the relative merits of 32 versus 64 bit computing, the transistor count is about a billion.

The x86 family of computers work using “registers.” A register is a place in the center of the CPU where a number can be held while other stuff gets done to it. If you want to add two numbers, you use a register. If you want to store the result, you probably use another register. They are absolutely crucial to the design of the x86, and most other widespread CPUs. (There are alternative architectures: stack based and accumulator based CPUs. But nobody buys them except for special applications.)

The x86 registers back in the time of the 8088 and 8086 computer were called:

AX The accumulator register, best for math.
BX The base address register, best for pointer references.
CX The counter register, best for loop counts.
DX The data register, used for multiply/divide operations, I/O operations, and indirect addressing.
SI The source index register, for memory-to-memory block operations, like copying data.
DI The destination index register, used for memory block operations, like SI.
BP The base pointer register, used to point to the base of the stack frame.
SP The stack pointer register, used to point to the current “top” of the stack.

There were some additional registers, like IP, flags, CS, DS, ES, and SS. But they weren’t (and still aren’t) general purpose registers, can’t participate in computations, and generally don’t get modified much by programs. Because the registers listed above are general purpose registers, they can be used for things not listed above. For example, if you need to store a temporary intermediate result while you are computing some other value, you can always jam it into the DI register. (or SI, or DX, or whatever you like). But not BP or SP.

So back in the day, when we had to carve our own computers out of rocks using nothing buy a bronze chisel and a wooden mallet, BP and SP were given over to the stack frame. That’s two registers out of eight — 25%, for you liberal arts majors — assigned to do the job of keeping track of the stack.

And now?

When the 80×86 family went 32-bit, the registers changed. The 32-bit registers were called “extended” registers, and so the names became: eax, ebx, ecx, edx, edi, esi, ebp, esp.

Now, with the advent of the 64-bit x86 family, the names have changed again: rax, rbx, rcx, rdx, rsi, rdi, rbp, rsp. But in addition, AMD added 8 new registers to the x64 architecture, and Intel followed suit.

This means that if you’re using a 32-bit CPU, or a 64-bit CPU in compatibility mode, you’re using 8 registers, still, and 2 of them are dedicated to the stack, still. (25%, still, for you liberal arts majors.) If you’re using a 64-bit CPU in 64-bit mode, then you’ve got 16 registers (that’s 12.5% for all you Art History grads).

So the stack frame, and the stack pointer, and this whole stack “thang” are pretty important.

What’s a buffer overflow attack?

So there’s a stack, and your subroutines use it. The most obvious way they can use it is to store a “return address” on it. If you have a subroutine called sub_1, it might have the stack set up like this:

    :
  ????
  ????
  1234 < -- SP

Where the SP (stack pointer) points to 1234. What’s 1234? Well, it’s the return address — an address is a location in computer memory — where it will send program control when it finishes doing whatever it is doing.

And if the sub_1 routine calls a sub_2 routine, the stack will look like:

    :
  ????
  ????
  1234
  1299 < -- SP

When the CPU executes a call instruction, it pushes the return address (of the instruction right after the call) onto the stack automatically. (And yes, on Intel platforms the stack always goes down instead of up. Don’t ask why, it doesn’t matter.)

When the sub_2 routine returns control to sub_1, it pops the stack and uses the address that it gets to branch back.

    :
  ????
  ????
  1234 < -- SP
  1299

Of course the 1299 is still down there — it doesn’t get erased. Instead, the SP adds 4 (4 is the size of a return address on a 32-bit [4 byte] CPU) and we go on. Very efficient, very fast, and very automatic with all the special instruction support.

Stack Frames, redux

When the program has local data it needs to store, the situation gets a little more complex. The caller pushes a return address, then the callee (the local subroutine) moves the stack pointer a little bit more, to make room. If sub_2 needs 12 bytes of storage, the preamble of the subroutine looks like this:

   push(ebp)
   ebp = esp
   esp = esp - 12

A preamble is a standard set of code that gets executed at the top of every function. It is generated automatically by the compiler — programmers don’t have to think about inserting it.

When it comes time to clean up and leave the subroutine, the corresponding code is the postamble and it does something like:

   esp = esp + 12
   pop ebp
   return

(In fact, this is wrong. But the effect is the same and it’s easier for you to understand.)

Before the subroutine is called, the stack looks like:

    :
  ????
  ????
  1234   < -- SP

Then when the call happens, the stack looks like:

    :
  ????
  ????
  1234
  1299  < -- SP

Then when the preamble runs, we get:

    :
  ????
  ????
  1234
  1299
   BP'  < -- BP
   ?
   ?
   ?     <-- SP (12 bytes = 3 x 32bit words of "stack frame" storage)

Note that the new BP is pointing to the place on the stack where the previous BP was stored.

So?

So there’s a stack frame, and it’s a little complex, and it’s sitting there on the stack. And in the middle of that stack is the “local data” the subroutine needs.

Sometimes the “local data” includes what is called a “buffer.” A buffer is just a space to store a bunch of characters in a row. For example, if the program is going to ask me for my name and address, it needs a place to store however-many characters go into a name and address.

If the buffer is 12 bytes long, I could enter my name as “Austin” and there would be not problem. But if I entered “Austin Hastings” (14 letters, with the space) the buffer would not be long enough to store the name. The problem is that if I am allowed to just enter any old thing, I might deliberately do something destructive. Suppose we are using our previous example:

    :
  ????
  ????
  1234
  1299
   BP'  < -- BP
   ?
   ?
   ?     <-- SP (12 bytes = 3 x 32bit words of "stack frame" storage)

If I type my name as “Austin Hastings”, the buffer overflows, like this:

    :
  ????
  ????
  1234
  1299
  ngs   < -- BP
  asti
  in H
  Aust  <-- SP

The old value of BP gets wiped out, and so the calling subroutine will have a bunch of bogus data (because it accesses all the local variables relative to where BP is pointing).

But if I type my name as “Austin Hastings.1215″ look what happens to the return address that is supposed to point to the subroutine that called us:

    :
  ????
  ????
  1234
  1215    -- look here!
  ngs.   < -- BP
  asti
  in H
  Aust  <-- SP

See? The return address got changed by me typing in a special, too-long name. This is called a “buffer overflow attack”, and if done right it can cause the subroutine to return control to a new program, written by the attacker.

Why can Microsoft solve the problem?

First, Microsoft writes Windows, the most attacked system in the world. And it writes the Microsoft C/C++ compiler, which is the tool of choice for compiling on Windows. Since anything Microsoft does will get copied by the Linux and Apple guys, this is a no-brainer. If Microsoft solves the problem, everyone will copy them and solve the problem for their own sites.

Okay, how can Microsoft solve the problem?

The solution to the “attack” part of the problem is to use two stacks. There are two registers already, and one of them rarely gets used (SP). So split them into two different stacks, and you eliminate the ability of an attacker to overwrite the return address (by separating the stack frame data from the “just-a-stack” data).

Details

Here’s what the previous example would look like with two stacks.

  "Frames":          "Stack":
      :                1234
    ????               EBP(1)  < -- SP(2)
    ????  <-- BP(2)    1299
    ....               EBP(2)  <-- SP(3)
    ....
    ....  <-- BP(3)

Notice that I’ve drawn the buffer as dots, and whatever was in the caller’s local data as question marks. Notice also that the important data — the frame pointers and return addresses — are in a totally different place from the stack frame data. This means that no buffer overflow attack can reach them.

That’s it?

Yep. That’s it.

Mind you, it’s an expensive it. But not a horribly expensive one. Microsoft, and the GCC folks, and Intel, and anybody else that writes a compiler, would have to come up with a transition plan to take us from one system to another. It would probably easier for Microsoft, since they can deliver their system (Windows) all at once. This recompile would have to happen for almost every problem, so it might take a while for total coverage. But the “most attacked” programs are things like email servers, web servers, the PHP interpreter, etc. There is a pretty short list of those, and getting them secured might take an hour and a half, if everyone would work together.

Any problems with this idea?

First, it doesn’t just magically make everything work. Some solutions, like W(+)E, try to do that. If you can just “solve it in hardware” then the problem goes away without messing around with that 23-year-old program that the boss uses but nobody has the source for.

Second, this approach doesn’t require anybody to buy a new computer. (But keep reading!) So unless Microsoft forces an upgrade with Windows 7 or 8 or whatever (and when have they ever forced an upgrade?) there’s no money in it for the CPU vendors.

Also, don’t forget that this only protects against buffer overrun attacks that try to force code execution. It won’t protect you from corruption of data that is in a stack frame buffer or local variable. Someone might try to rewrite their account balance, or change your password, or whatever.

Checksum the Stack Frame

There’s a solution for that, too. And it could involve new hardware. Just create a “checksum” value for the stack frame, and put it right at the end of each subroutine’s frame. That way, if an attacker overruns a buffer it will destroy the checksum.

Obviously I don’t mean a real checksum. What I mean is something unpredictable, unlikely to occur at random, and easily checked. Most of the registers wouldn’t work, since their values are generally predictable. It would make sense to create a dedicated register for this on the CPU — which would also allow Intel to make some new hardware — and push the “random” value on both the frame stack and the SP stack (for checking).

If you’re so damn smart, why aren’t you rich?

I used to do a lot of assembly programming. So I’ve always kind of wondered why this wasn’t obvious to everybody else. And it may well be that there’s some critical flaw in the logic. Probably having to do with interrupt-proofing the two-stack transition.

But maybe there’s not. Maybe it’s just obvious to me. In which case, I hereby demand that anyone who implements this must call it “Austin’s Device”. And if you implement the checksum idea, that shall be “Austin’s Other Device.”

But there’s an old joke about an Economics professor and one of his grad students walking across the quad. And the grad student says, “Look, professor! There’s a $100 bill lying on the ground.”

To which the professor replies, “Don’t be absurd. If there were really a $100 bill lying on the ground, someone would pick it up.”

Maybe you know what’s wrong with my clever device? If so, please let me know.

Things left unsaid…

Austin Hastings — Sat, 27 Jun 2009 23:15:18 +0000

In my copious spare time I’m working on a programming language called “Close.” It’s a C-like systems programming language targeted at the Parrot VM.

Now, I’m not a compiler guy. But I don’t have to be. Because Parrot comes with the Parrot Compiler Toolkit (PCT), a set of code that can render an Abstract Syntax Tree (AST) into executable code. The result is that I built a grammar using Perl6, then wrote some action methods hooked to the grammar that built the AST. Then I stood back and let PCT do it’s voodoo. The time between starting the project and now has been about 3 weeks, and I was busy with other stuff for 30 percent or so of that.

Okay, that’s cool. Parrot+PCT rocks, as Allison Randal points out. But it’s not my point. The reason I wrote Close was because people were still coding in PIR — Parrot’s version of assembly language — years after starting the project. D’oh! Everybody knows that the first thing you do with a new architecture is develop a C compiler.

Well, somebody had to do it. So I stepped up. I talked a little bit about it at the Parrot Virtual Machine Workshop last week in Pittsburgh, and some other people acted interested. As a result, it’s up on Google code at http://close.googlecode.com. But the key for me was a conversation I had with Uri Guttman in a hallway at YAPC10 (the conference to which the PVMW was attached).

I showed Uri this code:

void test_foreach()
{
	pmc the_list = new ResizableStringArray;
	push the_list, "alpha", "beta", "gamma", "delta", "omega";

	pmc new_list = new ResizableStringArray;
	int count = 0;

	foreach (str i: the_list) {
		push new_list, i;
		++count;
	}

	ok(count, 5, "f/e Iterate over each element");

	count = 0;
	int pass = 1;

	foreach (i : new_list) {
		if (i != the_list[count++]) {
			pass = 0;
		}
	}

	ok(pass, "f/e Same items, same order");
}

And Uri read through it, figured out what I was doing, and said, “Well, I don’t like your coding style. And I really don’t like single letter variable names, even inside a loop.”

And my first reaction was “Up yours, bud!” But I swallowed that, and then it hit me: I won!

The whole point of implementing Close was to stop writing code that looks like this:

.sub 'uniquereg' :method
    .param string rtype
    unless rtype goto err_nortype
    if rtype == 'v' goto reg_void
    .local string reg
    reg = 'P'
    $I0 = index 'Ss~Nn+Ii', rtype
    if $I0 < 0 goto reg_psin
    reg = substr 'SSSNNNII', $I0, 1
  reg_psin:
    reg = concat '$', reg
    .tailcall self.'unique'(reg)
  reg_void:
    .return ('')
  err_nortype:
    self.'panic'('rtype not set')
.end

And so what occurred to me was that I showed Uri some code that didn’t look like that. It looked like C. And so naturally he read it. And he commented on something that was relevant to him. Not “Gee, I can read your code!” but “Of course I can read your code — it’s C. But I don’t like your curly braces.”

Win. Win. Win.

Hearing what Uri did not say was the key.

Being a trust specialist

Austin Hastings — Mon, 27 Apr 2009 07:25:57 +0000

Most of the readers of this blog are CM specialists. Whether you’re a corporate CM librarian, or a build manager, you are focused on what the industry now calls “Application Lifecycle Management.” That’s an attempt to give a name to the collection of roles and functions we perform. It isn’t so much that the people are diversifying, as that the collection of tools that were all trying to fit under the “Configuration Management Tool” umbrella got too large. Change tracking? Sure, that’s a CM function. Version control? Yep. Requirements management? Well … okay. UML? Not so much. Trouble ticketing? Yeah, I guess. ITIL? Sure, why not?

Brad Appleton is, in my opinion, one of the all-around smart guys in the CM space. His focus for the last few years has been on Agile CM, but his writings are applicable to anybody doing software development, for a very broad definition of software - most of what he writes is applicable to almost any kind of intellectual property. And recently Brad has been doing some blogging about books on various flavors of ‘trust. ‘

Trust, to me, is one of those core values for most CM specialists. The fact is that CM is a simple job. There are certain requirements, and once you meet them you get to go home. In that way it’s a lot like being a system administrator: is the system up, is everything working? Okay, go home. Now admittedly it can be pretty hard to meet some of those requirements. That’s why I didn’t say it was an easy job - just a simple one. But that’s where trust comes in. Because whether your shop is an agile shop or not, the CM guys are more affected by trust than any single other thing.

Let me repeat that: CM guys are more affected by trust than any single other thing.

I’m a CM consultant. I get called in by shops that either don’t have CM and want to build a team, or by shops that already HAVE a CM team but have experienced enough CM failures that they don’t trust them any more. Since most companies obtain headcount by calling up a body shop and ordering it by the pound (kilos for the EU, I guess), I don’t usually get called first for CM team setups. It’s usually only after there are some bodies on site, when it turns out that CM specialists that are great at keeping an existing system going don’t have so much experience in starting a new system up from scratch. So there’s usually some kind of mess on the floor when I show up.

What does that mean for you? Well, it means that if you see me at your shop, you should make sure your resume is up to date. (Sorry, but it’s true.) It also means that you can keep me, or somebody like me, from showing up if you concentrate on building trust with your customers.

Brad’s first blog post about trust was on Covey’s “The Speed of Trust,” and in that book there are five “rings of trust” outlined. The second and third rings, or types, are interpersonal trust and organizational trust. That’s where I live, and if you’re smart it’s where you spend time each day maintaining your relationships with your customers.

It’s a pretty simple set of questions, but it makes all the difference.

Do the people you interact with understand what you need from them, and understand how to deliver it?
Do they understand what services you can provide to them, and how to request them?
Are all of these things practicable?
Do they perceive you as doing what is needed, on time, and without unnecessary oversight?
Do they perceive you as requesting things that are objective, and consistent, and in keeping with your and their function?
Do they believe the estimates and the explanations you give them?

If you can answer yes to all of those, you’ll never work with me. That’s because your customers trust you. If you can’t - if your customers distrust you - then as the level of trust decreases, the likelihood of a “fixer” being called in for your team goes up. In the credit card business, your interest rate goes up. In the CM business, the level of interest in working with you goes down.

If you’re a CM manager for a military systems contractor, then you’ve got a CM plan. And you’ve got a set of prescribed deliverables. And if the developers - software, firmware, systems engineers, whatever - know how to deliver each release to you, and it’s an automated, mechanical, simple chore, then you’re halfway there. If your other customers - the clients, and the development team that need to roll back to an earlier release - can submit a request and quickly get some kind of link, or bundle, or set of typing instructions, then you’re all the way done.

If you’re a build guy, or a release guy, or a deployment guy for a software team - agile or not - then the activities might be different but the questions aren’t. Does the team know how to get you to do a build, or cut a release, or deploy a package? Are there any bogus hoops they have to jump through? Have you built any passive-aggressive walls, saying “we want the development team to do this thing which they really ought to trust us to do but they overrode us that one time and now we’re bitter and whiny and nobody loves us”? Can a developer quickly get to the codeface of some old branch from six months ago?

I guess the short form is this. Are you doing your job? Are you making it as easy as possible for your customers to ask you to do your job for them? Because in my experience - and I’ll admit my experiences are all negative - if you aren’t doing your job, or if you’re making your customers pay some kind of “tax” (time, whining, paperwork, hoop-jumping, sloth, inefficiency) then you’re going to lose trust. And once you lose trust, you’ll see a little bureaucracy get started. And when that doesn’t work (and it rarely does) you’ll get to meet me. I’ll be the guy “we brought in to help gets things caught up.”

This isn’t a threat, quite. Because I don’t know you. It’s a reflection of the world as I see it, because I’m like a specialist doctor - nobody makes an appointment with a gastroenterologist to talk about how healthy they are. In my case, nobody calls me up and says, “We’d like to pay you to fly out here and sit in a conference room and live in a hotel for two weeks so that we can tell you how smoothly our build and deployment works!” (Man, I would LOVE that.) But what I get instead is, “We need you to come and fix our build and deployment procedures. Our developers don’t trust the CM team, and we’ve had a couple of blown deployments in the last few months.”

So anyway, I guess the upshot of this blog post - and this is much too depressing to be a CM Journal article, so it’ll stay a blog post - is that you need to make sure your house is in order, trust-wise. Go read one or more of the books on Brad’s list. Because CM teams seem far more Boolean than development teams. Developers can say “It’ll be done soon,” while CM’ers have to say “it’s not done.” And saying “it’s not done” leads pretty directly to “you’re not hired.”

Unit tests for MySQL scripts

Austin Hastings — Sun, 26 Apr 2009 01:07:09 +0000

Recently I had the opportunity to develop a unit testing framework for MySQL scripts in mostly-pure SQL. While I can’t share the code, I can certainly describe what we did, and why we did it. Hopefully it will be useful to you.

My client was reorganizing a development team into an agile mode in order to deal with an outsourcing disaster. It was one of those horror stories that starts out like “We were 3 years in to a 10 month project, when …” I was brought on to help ramp up their build and deployment processes, to get continuous integration and testing systems up, and to help mechanize their deployments.

The impetus for the MySQL unit test framework came from a short term contractor. The one “constant” in the otherwise highly-variable environment had been that the team wasn’t changing. Suddenly, that stopped being true. We had a handful of contractors in for a short term surge, and none of them were familiar with the application, or with SQL (it seemed). The offender was using the CI server as his syntax checker for database scripts. Yikes!

Luckily, the garbage he delivered was syntactically invalid. The mysql command-line client rejected the script and exited with an error status, which caused the CI script to log a failure, and … voila! Everyone credited me with doing a great job of implementing CI, because it was even checking the database stuff. Like a good consultant, I said “Yes. Of course.” Then I ran back to my desk, thinking “Oh, no! I’ve got to come up with a unit test framework for the MySQL stuff, pronto!” And I did.

Solutions

The software was a pretty complex web app, and the target environment was a mixture of Linux and Windows. As a result, when I looked at existing SQL unit test frameworks (dbUnit and sqlUnit, respectively) I had to reject them. They are decent products - and free - but the fact is that they expect certain things about their runtime environment that we couldn’t deliver. We had to build our own MySQL unit test framework, ideally in “pure” MySQL.

Fortunately for me, a nice guy named Giuseppe Maxia maintains a web site at datacharmer.org, and he has written a library of general-purpose MySQL code. (Called the General Purpose Stored Routine library, or gp_sr_lib.) Among the all this well-written, free, already debugged code was a set of testing routines. Thirty minutes into my development effort, I was already 80% done. Woo-hoo!

The GP library test routines are a good start on a mysqlUnit library, and I strongly recommend that if you need such a library you stop reading here and go download the code. Right now.

Rolling (y)our Own

There are a couple of gotchas that you’ll want to watch out for in your own unit testing library. First, the tests should be able to integrate into your CI engine. In particular, you need to decide on and implement some kind of failure mode - either fail when the first problem occurs, or fail after running all test cases. If you’re running the mysql client via some kind of exec call, be aware that while syntax errors will cause the client to exit with a failure code, other errors will not cause a failure. Instead, the client prints a diagnostic and returns zero. What’s more, the syntax does not support a deliberate abort from within the SQL engine.

In my own case, I chose to follow the model that Maxia has provided, and NOT fail as soon as a test fails. Rather, my code collected the results of all the tests that were run, and summarized them. In order to get “fail on error” behavior out of the mysql command line client, I coded a routine that would perform a “SELECT force_mysql_client_abort FROM no_such_table”. This definitely does cause the client to abort, at least until some developer creates a table called “no_such_table.” Then I wrapped the test cases in an “ignore failure” script that called the summarizer after all the cases had run, which in turn would call the abort routine if the error count was non-zero.

Another thing I chose to “fix” was the routine names. Maxia’s code includes both functions and procedures, but the names are not really in keeping with the style used in the various xUnit test frameworks. The library contains a procedure for asserting the existence of a database table. It is called as:

CALL check_table('db name', 'table name');

I opted to “embrace and extend” the library. I wrapped these types of checks into functions, like this:

set_database('db name'); assert_table_exists('table name'); assert_table_exists_in_db('other table', 'other db');

And in general, wherever there is a straight-ahead assert, I also added an “assert_not”. In terms of effect, there is no real difference in behavior between Maxia’s naming and my own. But I felt that the developers, who were using jUnit and nUnit, would be more comfortable learning a set of routines that had a similar structure and style.

Going Most of the Way: DDL

I tried to make a list of the various things that SQL scripts might do. The most obvious — and this probably accounted for two thirds or more of the scripts we had — is DDL statements. DDL (Data Definition Language) is that subset of SQL that relates to defining databases, tables, indexes, triggers, and the like. While the most frequently executed SQL statements are DML (Data Manipulation Language), DDL is where it’s at for most of the human-generated script files: CREATE TABLE, ADD INDEX, etc.

The great thing is that the assertions for these statements are trivial to write. Maxia’s library includes the CHECK_TABLE procedure, and some code for looking up procedures and functions. Adding other stuff, like keys and constraints, is pretty straightforward. By the end of the day you should have a long list of functions like assert_table_exists, assert_index_exists, assert_database_exists, and assert_column_has_type.

Deploy Early

At this point, you’re ready to go live. Except that you need to build a framework to run these tests. That’s going to depend a lot on your deployment process, and on your particular CI environment. Sorry. The simplest idea would be to write a script that matches the file names of test files, and runs them one at a time. Call that script from CI, and make sure the exit status is meaningful.

Forget the next couple of sections, because most of the database changes you’re likely to see are DDL. Developers and DBAs do a lot of adding fields, adding indexes, and sometimes adding tables. If you have just the existence assertions for each kind of object, your library will cover two-thirds or more of the statements being delivered. That’s not a trivial amount of test coverage for a day or two of work. Ship it! Your developers will need some training, and you’ll need to beat on the thing once it’s deployed to your CI servers to make sure that the errors are coming out correctly.

Sell to the Development Team

I recommend that you write some simple tests of things you know to be true. Confirm the positive cases first. Then sit down with a smart, involved developer and pair develop some unit tests for code he hasn’t written yet. Do it the way you want the developers to do it, including deciding how to handle assertions that don’t exist yet.

In our case, the DBAs had already established a naming convention for database update files. I chose to duplicate the file names, which would look like ####_description.sql, changing the .sql extension to .test.sql. This let the testing framework associate the test with the delivered change. When running a “from zero” deployment, the test framework would then run TEST-CHANGE-TEST and expect the first test to fail and the second to pass. This demonstrated that the test was meaningful for the change (remember, the change is supposed to repair a shortcoming that is demonstrated by the test).

Once you’ve got buy-in from your chosen developer, sit down and develop a presentation that the two of you can give. Point to specific examples of blown deployments as reasons for unit testing the SQL. Then get the coder to explain the available test assertions. Then explain the framework you’ve developed to integrate the testing library with CI, and explain what failures are going to look like. And don’t forget to point them at the documentation you’ve written on the project wiki. This should get your team on board, and now you can start looking for other assertions to write. You can also start tightening the screws on your framework. Make it so that no change can be delivered without at least one test.

Simple DML Checking

The next obvious category of SQL statements to check is simple DML. These are the CRUD verbs: INSERT, SELECT, UPDATE and DELETE. Except that SELECT can be pretty hard to check within SQL, so it’s best to ignore it for now in favor of the others. The problem with DML is knowing what to check for. If you insert a row, should you check that the ROW_COUNT() returns 1? Or should you query for the particular key of that row?

I’m pretty convinced that DML commands in scripts are going to be metadata related. That is, your application may be inserting customer records, or video titles, or whatever you track. But if a developer codes a DELETE, INSERT, or UPDATE into a deployable script, I think the intent is either to perform some kind of bulk fix (convert data fields, or eliminate nulls) or it is adjusting the “background” data — the list of postal codes, state and province names, etc. that your application relies on. In the case of the simple background data, if you added or updated it, you should check it explicitly: assert that for a specific key, the values are as expected.

Some examples of helpful functions would be assert_statement_affects_rowcount(prepared_statement, num_rows), assert_table_column_matches_count('table name', 'column', value, count), and their negatives. The way to do dynamic programming in MySQL is with prepared statements. You can construct a string containing a SQL statement, use PREPARE to assign it to an identifier, and then EXECUTE the identifier:

PREPARE stmt FROM CONCAT("SELECT COUNT(*) FROM ", table_name, " WHERE ", field_name, " = ?");

EXECUTE stmt USING value;

In the more complex case of a data format or type update, the update should be encoded in a stored procedure. This way you can unit test the stored procedure with sample data. Once the procedure is ready to go, write a script that calls the procedure - confident already that the procedure will work - and test the invocation by confirming random elements of the data have converted correctly.

Testing Result Sets

As mentioned, testing SELECT can be a real challenge. While it is easy to create an assert_row_count(num) to confirm that your query returned the right number of values, it can be much harder to create a function to assert that a particular ordering exists, or that the results are truly grouped by postal code.

The first thing to do is go ahead and write the assertion functions for number of rows returned by the query. That’s simple, straightforward, and it can stand in place of more complex stuff for a long time. You may not ever get around to writing the more challenging assertions — if the tests are designed well enough, the number of results may be meaningful enough. If you do need to write more, especially if the query is complex, pass the query as a string to your assertion, prepare it as a statement, and make sure the results go into a temporary table. Then you can write whatever horribly complex code you need against the temporary table, without having to worry about losing context.

Testing Triggers

Triggers are going to be harder to test than other pieces of code because of the diverse ways you can invoke them. For example, a DELETE trigger may need to be checked with a simple delete statement as well as with a CASCADE from a foreign key relationship. Beyond creating assertions for the existence and type of triggers (BEFORE/AFTER, INSERT/UPDATE/DELETE, etc.) there isn’t really any obvious set of assertions to write. It’s more a question of deciding how to test the effects of the triggers, and of documenting the ways the trigger can be invoked.

Conclusion

Unit testing, and test-first development, are well established best practices. But it’s easy to overlook the “small stuff” when developers are setting up frameworks for testing their code. Database updates are small, but when they go wrong they can take out the rest of the system. It’s important to apply the same techniques, and get the same value, to this part of your development. Now you can.

Three

Austin Hastings — Thu, 22 Jan 2009 11:52:13 +0000

As you can probably tell, my three LCD monitors are all present and accounted for. And yeah, before you ask, it ROCKS!

One very interesting “problem” is that none of the monitors I received shipped with a DVI cable. I have no idea why that is — all of the smaller LCD monitors I saw last summer came out of the box with both VGA and DVI cables, but these bigger, high-dollar products came with just VGA.

But that’s not a problem, because while the first monitor was a nice Hanns-G, which I plugged in to the VGA output from my laptop docking station, the other two are even nicer — they’re ASUS VW223-B monitors with “EzLink” (ASUS’s feature name for DisplayLink’s technology).

I found the two monitors on clearance from NewEgg.com. They were $159 plus NJ sales tax, call it $175 each. Apparently part of the clearance was getting rid of the products entirely, since the products are now shown as “Deactivated Item” on NewEgg’s site.

The DisplayLink/EzLink technology is an on-the-fly compression mechanism that sends video requests (probably just re-encoding Direct-X requests, but it might be something different) over USB. I wondered how well that technology would perform, but I was planning on buying a USB/VGA adapter device and finding out. Well, no need.

Both of the ASUS monitors are plugged in to USB, since they didn’t include a DVI adapter and since my Hanns-G is already taking up the VGA output port. And they work swimmingly. Not only is the display nice, but you can watch videos and other streaming media over the USB connection. I will admit that my limited experiment, watching three (different) videos on three different monitors at the same time, seemed to show some “snow” in the video images of the ASUS (USB linked) monitors. I’m guessing that this is a compression artifact, with the drivers sacrificing video in factor of speed. If that’s true, they’re doing it exactly right. The result was kind of like watching a slightly imperfect broadcast (over-the-air) television signal: a little fuzz, and some salt/pepper, but the underlying picture was clear.

The DisplayLink drivers installed smoothly, and once installed the driver had no problem picking up when I plugged in the third monitor. That little Windows “de-dum” sound for a new USB device, and I could instantly extend the desktop one more time. The monitors can be controlled from the taskbar, including specifically a rotation option. Also, the monitors appear in the standard Windows Display control panel, so you can specify location and resolution.

As you can see in the above image, setting the rotation causes the correct data to be reflected back to windows. And you’ll have to take my word for it, but you can get a LOT of lines of code onto a 22″ monitor in portrait mode. Here’s the bottom of a Scite window, where the top of the window is showing top-of-file:

Yes, that’s 104 lines of source code on a single screen. Given that subroutines and methods are supposed to fit “on a single page,” I may have discovered the ultimate solution to all refactoring problems.

This is so cool!

One thing worth pointing out is that the monitors themselves do not support rotation. They come with the kind of cheap plastic stand that you would expect. Instead, I bought a third-party monitor stand from the Ergotech Group. They don’t have an on-line store, but rely on other vendors for retail sales. I found a bunch of places where I could choose to pay about a million dollars for that stand, and then I found PC Connection, where $250 seemed like a bargain.

All told, the monitors ($522) and the stand ($267) cost me $790. Sitting here, beginning to develop retinal cancer from the massive volume of monitor-light being blasted at my eyes, it was totally worth it.

Getting L A R G E R

Austin Hastings — Fri, 16 Jan 2009 08:22:53 +0000

I just spent a bunch of time out west (which, since I live on the east coast, isn’t saying much) working with a client trying to do agile development. I have a kind of gentlemen’s agreement with the client that I won’t write anything that makes them look too bad. Presumably, they’ll be kind enough to do the same thing for me :).

That said, I want to talk about something that I absolutely cannot stand about the development environment out there: They have really big monitors, usually two per pairing station. And I don’t.

It’s a little bit surprising how easy it was to get used to working with multiple monitors. In fact, towards the end of the engagement I got moved in to a cubicle that had three monitors in it - one for the old user’s laptop, one for a desktop that was sitting there, and one small one unused. Since I was bringing two monitors with me, that took me up to five monitors. Sadly, I couldn’t talk the IT guys into buying the Lenovo advanced docking station that supported a plug-in video controller, or I’m sure I could have driven all five monitors off my laptop.

Now, you might be asking “what on earth could you do with five monitors?” And, you know, I’m not entirely sure myself. But I would surely have loved to find out. In the meantime, I got away with using Synergy to share my laptop’s keyboard and mouse with another computer, and then drove three monitors from my laptop with a USB video adapter.

What can I do with four monitors? One of them was my designated remote desktop box, for connecting to the Windows servers the team was using. The other three? Well, whatever I wanted.

I almost always had Firefox open in one monitor, maximized. Because I was almost always working on something that required frequent reference to online forums or documentation. CM consultants never spend much time in the “well documented, easy to configure” part of the system. Go figure.

Sometimes I had email open in another monitor, but more often I had a text editor maximized. And then I stuck a couple of shell windows (MSYS, Powershell, Cygwin, rxvt, whatever it takes) in the other monitor. And then I silently wished that I could have gotten that fifth monitor up, because sometimes you need to RDP into two different remote servers, in order to test a deployment script.

So, yeah. I’m addicted to having multiple monitors. LOTS of them. And my addiction is growing. Two monitors, which I’m using now, just isn’t enough anymore. So I’ve taken steps. I’m ordering more monitors. And a monitor stand to hang them from, because that was one thing that I didn’t have out west. And this time, they’re going to be BIG.

Apparently, Alienware hasn’t released their promised 3′-wide ubermonitor, but Panasonic has got a 103″ screen in production. Sadly, I can’t afford one. So I’m going to settle for 22″ LCD monitors.

Now, you’re probably saying “Wait, why not get two 24 or 30 inch monitors?” And there are three answers. One, I like having a “main” screen in front of me. Two monitors would put the main screen at an angle (bad) or put the main screen in front and require twisting my neck in one direction half the time (bad). So three is the right number. Second, the native resolutions for the big displays is too large for my laptop to drive, and is also too large for a USB video adapter to drive, and is also too large for a Matrox TripleHead2Go adapter to drive. So 1650×1050 or 1650×1200 is the right size. And third, they don’t make three-across monitor mounting brackets quite that big yet.

Two Conferences

Austin Hastings — Wed, 17 Sep 2008 02:15:52 +0000

Austin Hastings (that’s me) will be speaking at two conferences coming up in October. Both presentations will be focusing on LDM.

First, there’s the Telelogic User Group Conference 2008, in Austin, Texas. That’s obviously focused on Telelogic, and on CM Synergy and Telelogic Change, or whatever it’s called this week. Since I spent a fair amount of time during the initial implementation of LDM wishing that I could use the kind of one-off ad hoc queries that Change is so good at, I suppose it’s appropriate.

The second conference is CMconf.DE, in Munich, Germany. Sadly, it will be a few weeks after Oktoberfest — otherwise there would be no hope at all of getting hotel space, I suppose. I’ll just have to wander around looking for any leftover barrels of beer, slightly-used lederhosen, or oompa bands that I can scrounge up. The theme is the same — presenting LDM and some other approaches to database CM.

In both cases, the time constraints are really biting. LDM needs an understanding of the other ways to do (fail at) database CM before I can really talk about it, and I’m pretty sure that most people haven’t thought beyond one way, if that many. So I’ll be speaking really, really fast, and inviting attendees to stick around afterwards to talk about it.

Anyway, that’s two continents in a single month — not bad, if I do say so. If I could just find a conference in India, China, or Israel, I could sweep the northern hemisphere.

Gotcha: XML include processing in CruiseControl.NET

Austin Hastings — Tue, 24 Jun 2008 16:24:36 +0000

Release 1.4 of CruiseControl.NET includes a new Configuration Preprocessor. The great thing is that this provides a C preprocessor-like macro syntax, so projects can be made using relatively powerful templates.

The bad thing is that it’s new, and so there are some gotcha’s laying around. I just found one:

I defined a ccnet.config file structured like this:

  
  
  
  
    
      
    
  
  
  

That worked fine, and I could see how to rubber-stamp that project template to all my various continuous-integration areas and other workspaces.

But then I got clever, and decided to separate out the template from the local configuration.

  
  

  
  

  
  

By unfortunate coincidence, I had to reboot my laptop at this point — I kicked the power strip under the desk, removing power to the docking station. The Lenovo I’m using (client provided) doesn’t handle this at all, so it was restart time. When I came back up, the CruiseControl.NET service wouldn’t start.

Debugging is simple and easy with the “ccnet.exe” command line utility. Just run
ccnet.exe -config:ccnet.config -validate

The error, though, was incomprehensible:

C:\Program Files\CruiseControl.NET\server>ccnet -config:ccnet.config -validate
CruiseControl.NET Server 1.4.0.3400 -- .NET Continuous Integration Server
Copyright c 2008 ThoughtWorks Inc.  All Rights Reserved.
.NET Runtime Version: 2.0.50727.1433    Image Runtime Version: v2.0.50727
OS Version: Microsoft Windows NT 5.1.2600 Service Pack 3        Server locale: e
n-US

[CCNet Server:DEBUG] The trace level is currently set to debug.  This will cause
 CCNet to log at the most verbose level, which is useful for setting up or debug
ging the server.  Once your server is running smoothly, we recommend changing th
is setting in C:\Program Files\CruiseControl.NET\server\ccnet.exe.config to a lo
wer level.
[CCNet Server:INFO] Reading configuration file "C:\Program Files\CruiseControl.NET\server\ccnet.config"
[CCNet Server:ERROR] Exception: The configuration file contains invalid xml: C:\Program Files\CruiseControl.NET\server\ccnet.config
----------
ThoughtWorks.CruiseControl.Core.Config.ConfigurationException: The configuration file contains invalid xml: C:\Program Files\CruiseControl.NET\server\ccnet.config ---> System.Xml.XmlException: 'cb' is an undeclared namespace. Line 1, position 2.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.Throw(String res, String arg, Int32 lineNo, Int32 linePos)
   at System.Xml.XmlTextReaderImpl.LookupNamespace(NodeData node)
   at System.Xml.XmlTextReaderImpl.ElementNamespaceLookup()
   at System.Xml.XmlTextReaderImpl.ParseAttributes()
   at System.Xml.XmlTextReaderImpl.ParseElement()
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XPath.XPathDocument.LoadFromReader(XmlReader reader, XmlSpace space)
   at System.Xml.XPath.XPathDocument..ctor(Stream stream)
   at ThoughtWorks.CruiseControl.Core.Config.Preprocessor.ConfigPreprocessorEnvironment.push_include(String href)
   at ThoughtWorks.CruiseControl.Core.Config.Preprocessor.ConfigPreprocessor.PreProcess(XmlReader input, XmlWriter output, PreprocessorUrlResolver resolver, Uri input_uri)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.CreateXmlValidatingLoader(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.AttemptLoadConfiguration(FileInfo configFile)
   --- End of inner exception stack trace ---
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.AttemptLoadConfiguration(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.DefaultConfigurationFileLoader.Load(FileInfo configFile)
   at ThoughtWorks.CruiseControl.Core.Config.FileConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.Config.FileWatcherConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.Config.CachingConfigurationService.Load()
   at ThoughtWorks.CruiseControl.Core.CruiseServer..ctor(IConfigurationService configurationService, ProjectIntegratorListFactory projectIntegratorListFactory, IProjectSerializer projectSerializer)
   at ThoughtWorks.CruiseControl.Core.CruiseServerFactory.Create(Boolean remote, String configFile)
   at ThoughtWorks.CruiseControl.Core.ConsoleRunner.Run()
   at ThoughtWorks.CruiseControl.Console.ConsoleMain.Main(String[] args)
----------

Even worse, the error message is presented in white text on a glaring red background, so as to be all but unreadable. Looking at it now, in retrospect (and black & white), I can see clues that I missed the first time. Specifically, the string “push_include.”

Anyway, the problem turned out to be the way the Configuration Processor deals with XML includes. It is not a C preprocessor — it includes subtrees, not text.

The answer turns out to be in the included file. References in the included file to the “cb:” XML namespace have to be accompanied in that file by some definition. Adding the same xmlns= statement does the trick:



xmlns:cb="urn:ccnet.config.builder"  name = "PROJECT"  >

Problem solved. Now let the rubber-stamping begin!

Microsoft's take on database change management

Austin Hastings — Tue, 27 May 2008 13:58:34 +0000

As a sort of follow-on to my post on doing Database CM Three Ways, I give you this presentation from Microsoft field rep Jim Blizzard.

The presentation video goes over using the entire suite of Microsoft tools — go figure, right? But it is a good, solid introduction to what I call the “Computed List of Differences” approach to database CM. Using the Visual Studio environment, you track the database source code as you are making changes. Then, you can compare the “baseline” source code, or perhaps the live database schema, against what you have in your local workspace. The differences are computed automatically by the tool.

This presumes, of course, that you are using MS products across the board. The demo doesn’t show what would happen if your database were Oracle, or MySql. (Though I think I can guess… ) And it ignores the issue of program data that is stored in the database itself, instead of in the schema. Suppose you have the classic lookup table listing all the US states and territories. Then suppose we add a new entry to that table. The 51st state, or a new US protectorate — “IQ” anyone? Making the database change is trivial, of course, but I don’t think the schema diff tool would catch it.

At any rate, the video is the state of the art in a lot of ways. If you’re interested in DBCM, you need to have a look.