Brad Appleton is, in my opinion, one of the all-around smart guys in the CM space. His focus for the last few years has been on Agile CM, but his writings are applicable to anybody doing software development, for a very broad definition of software – most of what he writes is applicable to almost any kind of intellectual property. And recently Brad has been doing some blogging about books on various flavors of ‘trust. ‘

Trust, to me, is one of those core values for most CM specialists. The fact is that CM is a simple job. There are certain requirements, and once you meet them you get to go home. In that way it’s a lot like being a system administrator: is the system up, is everything working? Okay, go home. Now admittedly it can be pretty hard to meet some of those requirements. That’s why I didn’t say it was an easy job – just a simple one. But that’s where trust comes in. Because whether your shop is an agile shop or not, the CM guys are more affected by trust than any single other thing.

Let me repeat that: CM guys are more affected by trust than any single other thing.

I’m a CM consultant. I get called in by shops that either don’t have CM and want to build a team, or by shops that already HAVE a CM team but have experienced enough CM failures that they don’t trust them any more. Since most companies obtain headcount by calling up a body shop and ordering it by the pound (kilos for the EU, I guess), I don’t usually get called first for CM team setups. It’s usually only after there are some bodies on site, when it turns out that CM specialists that are great at keeping an existing system going don’t have so much experience in starting a new system up from scratch. So there’s usually some kind of mess on the floor when I show up.

What does that mean for you? Well, it means that if you see me at your shop, you should make sure your resume is up to date. (Sorry, but it’s true.) It also means that you can keep me, or somebody like me, from showing up if you concentrate on building trust with your customers.

Brad’s first blog post about trust was on Covey’s “The Speed of Trust,” and in that book there are five “rings of trust” outlined. The second and third rings, or types, are interpersonal trust and organizational trust. That’s where I live, and if you’re smart it’s where you spend time each day maintaining your relationships with your customers.

It’s a pretty simple set of questions, but it makes all the difference.

1. Do the people you interact with understand what you need from them, and understand how to deliver it?
2. Do they understand what services you can provide to them, and how to request them?
3. Are all of these things practicable?
4. Do they perceive you as doing what is needed, on time, and without unnecessary oversight?
5. Do they perceive you as requesting things that are objective, and consistent, and in keeping with your and their function?
6. Do they believe the estimates and the explanations you give them?

If you can answer yes to all of those, you’ll never work with me. That’s because your customers trust you. If you can’t – if your customers distrust you – then as the level of trust decreases, the likelihood of a “fixer” being called in for your team goes up. In the credit card business, your interest rate goes up. In the CM business, the level of interest in working with you goes down.

If you’re a CM manager for a military systems contractor, then you’ve got a CM plan. And you’ve got a set of prescribed deliverables. And if the developers – software, firmware, systems engineers, whatever – know how to deliver each release to you, and it’s an automated, mechanical, simple chore, then you’re halfway there. If your other customers – the clients, and the development team that need to roll back to an earlier release – can submit a request and quickly get some kind of link, or bundle, or set of typing instructions, then you’re all the way done.

If you’re a build guy, or a release guy, or a deployment guy for a software team – agile or not – then the activities might be different but the questions aren’t. Does the team know how to get you to do a build, or cut a release, or deploy a package? Are there any bogus hoops they have to jump through? Have you built any passive-aggressive walls, saying “we want the development team to do this thing which they really ought to trust us to do but they overrode us that one time and now we’re bitter and whiny and nobody loves us”? Can a developer quickly get to the codeface of some old branch from six months ago?

I guess the short form is this. Are you doing your job? Are you making it as easy as possible for your customers to ask you to do your job for them? Because in my experience – and I’ll admit my experiences are all negative – if you aren’t doing your job, or if you’re making your customers pay some kind of “tax” (time, whining, paperwork, hoop-jumping, sloth, inefficiency) then you’re going to lose trust. And once you lose trust, you’ll see a little bureaucracy get started. And when that doesn’t work (and it rarely does) you’ll get to meet me. I’ll be the guy “we brought in to help gets things caught up.”

This isn’t a threat, quite. Because I don’t know you. It’s a reflection of the world as I see it, because I’m like a specialist doctor – nobody makes an appointment with a gastroenterologist to talk about how healthy they are. In my case, nobody calls me up and says, “We’d like to pay you to fly out here and sit in a conference room and live in a hotel for two weeks so that we can tell you how smoothly our build and deployment works!” (Man, I would LOVE that.) But what I get instead is, “We need you to come and fix our build and deployment procedures. Our developers don’t trust the CM team, and we’ve had a couple of blown deployments in the last few months.”

So anyway, I guess the upshot of this blog post – and this is much too depressing to be a CM Journal article, so it’ll stay a blog post – is that you need to make sure your house is in order, trust-wise. Go read one or more of the books on Brad’s list. Because CM teams seem far more Boolean than development teams. Developers can say “It’ll be done soon,” while CM’ers have to say “it’s not done.” And saying “it’s not done” leads pretty directly to “you’re not hired.”

My client was reorganizing a development team into an agile mode in order to deal with an outsourcing disaster. It was one of those horror stories that starts out like “We were 3 years in to a 10 month project, when …” I was brought on to help ramp up their build and deployment processes, to get continuous integration and testing systems up, and to help mechanize their deployments.

The impetus for the MySQL unit test framework came from a short term contractor. The one “constant” in the otherwise highly-variable environment had been that the team wasn’t changing. Suddenly, that stopped being true. We had a handful of contractors in for a short term surge, and none of them were familiar with the application, or with SQL (it seemed). The offender was using the CI server as his syntax checker for database scripts. Yikes!

Luckily, the garbage he delivered was syntactically invalid. The mysql command-line client rejected the script and exited with an error status, which caused the CI script to log a failure, and … voila! Everyone credited me with doing a great job of implementing CI, because it was even checking the database stuff. Like a good consultant, I said “Yes. Of course.” Then I ran back to my desk, thinking “Oh, no! I’ve got to come up with a unit test framework for the MySQL stuff, pronto!” And I did.

Solutions

The software was a pretty complex web app, and the target environment was a mixture of Linux and Windows. As a result, when I looked at existing SQL unit test frameworks (dbUnit and sqlUnit, respectively) I had to reject them. They are decent products – and free – but the fact is that they expect certain things about their runtime environment that we couldn’t deliver. We had to build our own MySQL unit test framework, ideally in “pure” MySQL.

Fortunately for me, a nice guy named Giuseppe Maxia maintains a web site at datacharmer.org, and he has written a library of general-purpose MySQL code. (Called the General Purpose Stored Routine library, or gp_sr_lib.) Among the all this well-written, free, already debugged code was a set of testing routines. Thirty minutes into my development effort, I was already 80% done. Woo-hoo!

The GP library test routines are a good start on a mysqlUnit library, and I strongly recommend that if you need such a library you stop reading here and go download the code. Right now.

Rolling (y)our Own

There are a couple of gotchas that you’ll want to watch out for in your own unit testing library. First, the tests should be able to integrate into your CI engine. In particular, you need to decide on and implement some kind of failure mode – either fail when the first problem occurs, or fail after running all test cases. If you’re running the mysql client via some kind of exec call, be aware that while syntax errors will cause the client to exit with a failure code, other errors will not cause a failure. Instead, the client prints a diagnostic and returns zero. What’s more, the syntax does not support a deliberate abort from within the SQL engine.

In my own case, I chose to follow the model that Maxia has provided, and NOT fail as soon as a test fails. Rather, my code collected the results of all the tests that were run, and summarized them. In order to get “fail on error” behavior out of the mysql command line client, I coded a routine that would perform a “SELECT force_mysql_client_abort FROM no_such_table”. This definitely does cause the client to abort, at least until some developer creates a table called “no_such_table.” Then I wrapped the test cases in an “ignore failure” script that called the summarizer after all the cases had run, which in turn would call the abort routine if the error count was non-zero.

Another thing I chose to “fix” was the routine names. Maxia’s code includes both functions and procedures, but the names are not really in keeping with the style used in the various xUnit test frameworks. The library contains a procedure for asserting the existence of a database table. It is called as:

CALL check_table('db name', 'table name');


I opted to “embrace and extend” the library. I wrapped these types of checks into functions, like this:


set_database('db name');
assert_table_exists('table name');
assert_table_exists_in_db('other table', 'other db');


And in general, wherever there is a straight-ahead assert, I also added an “assert_not”. In terms of effect, there is no real difference in behavior between Maxia’s naming and my own. But I felt that the developers, who were using jUnit and nUnit, would be more comfortable learning a set of routines that had a similar structure and style.

Going Most of the Way: DDL

I tried to make a list of the various things that SQL scripts might do. The most obvious — and this probably accounted for two thirds or more of the scripts we had — is DDL statements. DDL (Data Definition Language) is that subset of SQL that relates to defining databases, tables, indexes, triggers, and the like. While the most frequently executed SQL statements are DML (Data Manipulation Language), DDL is where it’s at for most of the human-generated script files: CREATE TABLE, ADD INDEX, etc.

The great thing is that the assertions for these statements are trivial to write. Maxia’s library includes the CHECK_TABLE procedure, and some code for looking up procedures and functions. Adding other stuff, like keys and constraints, is pretty straightforward. By the end of the day you should have a long list of functions like assert_table_exists, assert_index_exists, assert_database_exists, and assert_column_has_type.

Deploy Early

At this point, you’re ready to go live. Except that you need to build a framework to run these tests. That’s going to depend a lot on your deployment process, and on your particular CI environment. Sorry. The simplest idea would be to write a script that matches the file names of test files, and runs them one at a time. Call that script from CI, and make sure the exit status is meaningful.

Forget the next couple of sections, because most of the database changes you’re likely to see are DDL. Developers and DBAs do a lot of adding fields, adding indexes, and sometimes adding tables. If you have just the existence assertions for each kind of object, your library will cover two-thirds or more of the statements being delivered. That’s not a trivial amount of test coverage for a day or two of work. Ship it! Your developers will need some training, and you’ll need to beat on the thing once it’s deployed to your CI servers to make sure that the errors are coming out correctly.

Sell to the Development Team

I recommend that you write some simple tests of things you know to be true. Confirm the positive cases first. Then sit down with a smart, involved developer and pair develop some unit tests for code he hasn’t written yet. Do it the way you want the developers to do it, including deciding how to handle assertions that don’t exist yet.

In our case, the DBAs had already established a naming convention for database update files. I chose to duplicate the file names, which would look like ####_description.sql, changing the .sql extension to .test.sql. This let the testing framework associate the test with the delivered change. When running a “from zero” deployment, the test framework would then run TEST-CHANGE-TEST and expect the first test to fail and the second to pass. This demonstrated that the test was meaningful for the change (remember, the change is supposed to repair a shortcoming that is demonstrated by the test).

Once you’ve got buy-in from your chosen developer, sit down and develop a presentation that the two of you can give. Point to specific examples of blown deployments as reasons for unit testing the SQL. Then get the coder to explain the available test assertions. Then explain the framework you’ve developed to integrate the testing library with CI, and explain what failures are going to look like. And don’t forget to point them at the documentation you’ve written on the project wiki. This should get your team on board, and now you can start looking for other assertions to write. You can also start tightening the screws on your framework. Make it so that no change can be delivered without at least one test.

Simple DML Checking

The next obvious category of SQL statements to check is simple DML. These are the CRUD verbs: INSERT, SELECT, UPDATE and DELETE. Except that SELECT can be pretty hard to check within SQL, so it’s best to ignore it for now in favor of the others. The problem with DML is knowing what to check for. If you insert a row, should you check that the ROW_COUNT() returns 1? Or should you query for the particular key of that row?

I’m pretty convinced that DML commands in scripts are going to be metadata related. That is, your application may be inserting customer records, or video titles, or whatever you track. But if a developer codes a DELETE, INSERT, or UPDATE into a deployable script, I think the intent is either to perform some kind of bulk fix (convert data fields, or eliminate nulls) or it is adjusting the “background” data — the list of postal codes, state and province names, etc. that your application relies on. In the case of the simple background data, if you added or updated it, you should check it explicitly: assert that for a specific key, the values are as expected.

Some examples of helpful functions would be assert_statement_affects_rowcount(prepared_statement, num_rows), assert_table_column_matches_count('table name', 'column', value, count), and their negatives. The way to do dynamic programming in MySQL is with prepared statements. You can construct a string containing a SQL statement, use PREPARE to assign it to an identifier, and then EXECUTE the identifier:


PREPARE stmt
FROM
CONCAT("SELECT COUNT(*) FROM ", table_name, " WHERE ", field_name, " = ?");



EXECUTE stmt USING value;


In the more complex case of a data format or type update, the update should be encoded in a stored procedure. This way you can unit test the stored procedure with sample data. Once the procedure is ready to go, write a script that calls the procedure – confident already that the procedure will work – and test the invocation by confirming random elements of the data have converted correctly.

Testing Result Sets

As mentioned, testing SELECT can be a real challenge. While it is easy to create an assert_row_count(num) to confirm that your query returned the right number of values, it can be much harder to create a function to assert that a particular ordering exists, or that the results are truly grouped by postal code.

The first thing to do is go ahead and write the assertion functions for number of rows returned by the query. That’s simple, straightforward, and it can stand in place of more complex stuff for a long time. You may not ever get around to writing the more challenging assertions — if the tests are designed well enough, the number of results may be meaningful enough. If you do need to write more, especially if the query is complex, pass the query as a string to your assertion, prepare it as a statement, and make sure the results go into a temporary table. Then you can write whatever horribly complex code you need against the temporary table, without having to worry about losing context.

Testing Triggers

Triggers are going to be harder to test than other pieces of code because of the diverse ways you can invoke them. For example, a DELETE trigger may need to be checked with a simple delete statement as well as with a CASCADE from a foreign key relationship. Beyond creating assertions for the existence and type of triggers (BEFORE/AFTER, INSERT/UPDATE/DELETE, etc.) there isn’t really any obvious set of assertions to write. It’s more a question of deciding how to test the effects of the triggers, and of documenting the ways the trigger can be invoked.

Conclusion

Unit testing, and test-first development, are well established best practices. But it’s easy to overlook the “small stuff” when developers are setting up frameworks for testing their code. Database updates are small, but when they go wrong they can take out the rest of the system. It’s important to apply the same techniques, and get the same value, to this part of your development. Now you can.

First, there’s the Telelogic User Group Conference 2008, in Austin, Texas. That’s obviously focused on Telelogic, and on CM Synergy and Telelogic Change, or whatever it’s called this week. Since I spent a fair amount of time during the initial implementation of LDM wishing that I could use the kind of one-off ad hoc queries that Change is so good at, I suppose it’s appropriate.

The second conference is CMconf.DE, in Munich, Germany. Sadly, it will be a few weeks after Oktoberfest — otherwise there would be no hope at all of getting hotel space, I suppose. I’ll just have to wander around looking for any leftover barrels of beer, slightly-used lederhosen, or oompa bands that I can scrounge up. The theme is the same — presenting LDM and some other approaches to database CM.

In both cases, the time constraints are really biting. LDM needs an understanding of the other ways to do (fail at) database CM before I can really talk about it, and I’m pretty sure that most people haven’t thought beyond one way, if that many. So I’ll be speaking really, really fast, and inviting attendees to stick around afterwards to talk about it.

Anyway, that’s two continents in a single month — not bad, if I do say so. If I could just find a conference in India, China, or Israel, I could sweep the northern hemisphere.

These guys don’t seem to have any more idea than most about how to manage database changes, but their product (or at least, the pdf of the glossy brochure) looks to have some really good tracking built in for deployments in both i-space and Windows-space. They claim that (for a small additional fee) you can track deployments all the way to the customer.

This is typical of a mainframe approach. The mainframe guys don’t have bajillions of PC customers, they have tens, or hundreds, or thousands of mainframe customers. Those are small enough numbers that it makes sense to talk about tracking them individually. It isn’t surprising, therefore, that mainframers have a more pragmatic approach to tracking deployments.

Thinking about how I could get my hands on a copy of MatchPoint (when I don’t own an IBM i-series) has led me to think some more about how to review a CM tool. I want to be able to write CM tool reviews, but I don’t want to write the kind of bunkum that you get from most 1500-word specials.

If you have any hands-on experience with MatchPoint, I’d love to hear from you — to get some real world feedback on their glossy brochure, if nothing else. And if you have ideas on how to review a CM tool, well, I’ll write something up in a separate post. Hold that thought.

The simplest capability challenge from an SCM standpoint is SCM capability. Can you, or can your team, effectively manage the changes and/or configuration of your project? It’s hard to diagnose a lack of capability, because the team members are too busy dealing with the symptoms—each problem that comes up has a simple, direct solution. As the saying goes, “It’s hard to remember that you set out to drain the swamp when you’re neck deep in alligators.” But if you are struggling with failed builds or missing source code, or if the QA department is wondering why a bug they marked as ‘fixed’ has reappeared in the product, or if you are poring through someone’s email archive looking for information that should be available from the repository, then you have a capability problem.

More likely, though, is that you have some basic CM up and running, and you are being challenged by a new project. Maybe there is a new technology that needs to be integrated with your build system, or a new office that you have to support. These cases are easier to detect, since they occur as a result of a change to the project itself. The root cause of the capability challenge may come from another dimension listed in this series, such as geography, technology, or schedule, but if you cannot respond or don’t know how to respond, then part of the problem is capability.

A good technique for basic CM capability problems is Subject Matter Expert. (Note that the author is a CM consultant, and so benefits from giving this advice.) Hire or appoint someone to deal with CM issues, either to solve a particular problem or to provide ongoing expertise for your organization.

Beyond just CM capability, other capability challenges may impact your CM team. If management decides to outsource some or all of your next project, you will face capability issues—the new team may not know how to use your tool suite, or may not know the technologies your team uses. They certainly won’t know the internal details of your project. Depending on the structure of your project, you may find that you have hardware capability issues—not enough network bandwidth, not enough build capacity, not enough disk space. Knowledge acquisition is generally simple—spend money on training. Likewise, hardware capacity problems are solvable with cash.

Most other capability challenges are part of a larger problem, likely one that falls mostly along another dimension in the list. If executive management decides to enter a new market, you may face regulatory or human language issues. Restructuring, mergers, and joint development can lead to location of responsibility, culture, geography, and knowledge retention issues. Not all ‘can we?’ problems are capability problems. But nearly all problems will have a capability component. Geographically distributed development calls for networking multiple servers; adding a new technology requires training for the developers; and regulatory compliance requires updating or adding formal processes. If you don’t already have them, you have to acquire the resources to solve the problem.

(This was published as part of “Dimensions of SCM Challenge: Capability and Finances” in CM Journal, Vol. 6, No. 2, February 2008.)

Changing or unknown requirements can be a huge source of problems for a development team. Fortunately, configuration management is designed to support exactly this development problem. Books and tools abound that address the issues of managing changes, documenting and tracing requirements, and ensuring that the true needs of the eventual customers are met by the project. Requirements elicitation and elucidation, change management, and traceability tools are all available in the commercial market, and Agile development methods can ensure that customer representatives are involved from the very beginning. Tracking changes to requirements is not the same as tracking changes to software, but nearly every CM tool vendor offers an extension or plug-in for requirements management. Similarly, the requirements management tool vendors offer change tracking systems, as well as integrations with CM tools.

The most basic requirements challenge is knowing that a requirement exists at all. When a project transitions from a “clever idea” to a formal product, or switches from development to maintenance, the developer(s) will begin to get change requests and feature suggestions from a multitude of new sources. Keeping track of the ideas, and which ones are approved, is an important first step. Frequently this is implemented using the team’s change tracking tool. As the difference between requirements and change requests becomes more clear, the team can move to Formal Requirements Management.

The next source of requirements challenge is simple: a requirement to do something really, really hard. There isn’t much that CM can do to help directly with this kind of requirement. A good requirements management tool will, however, be able to predict the impact of the requirement—which parts of the project are affected by the requirement, and so likely to change. In some cases a single difficult requirement is really a Technical Complexity challenge (solve the Traveling Salesman problem), or a Technological Diversity challenge (integrate with a decades-old legacy system), rather than a requirements challenge. Please refer to the appropriate article, when they come out.

One common form of the “hard to do” problem is a requirement that the development team has never faced before. If a team that is accustomed to performance requirements is suddenly faced with a peak power requirement, for example, the team may get “stuck” with little or no idea of how to proceed. This may genuinely be a hard problem, but more likely is that the solution already exists. The solution in this case may be a new technology, especially if the requirement is being driven by managerial enthusiasm. Beware of hearing “this is the new state of the art” in meetings, and expect to deal with Technological Diversity challenges. When a team is facing an entirely new category of requirement, Subject Matter Expert is a good bet for a solution.

Another source of heretofore-unseen requirements is organizational change. When an organization is being divided, or when two organizations are being merged, some new and strange requirements can appear. (“Integrate our product with that other product over there. Make it seamless!”) Beware of other Organizational Structure challenges, obviously. Sending an envoy from one team to another makes sense as a basic Subject Matter Expert approach, but identifying someone to be sent to another team can also identify them as a target for layoffs—there aren’t a lot of easy answers.

When the challenge stems from the number of requirements, the complexity of the requirements, or from the need to verify the requirements, the answer is going to be Requirements Management. (It seems a little shady to be classifying an entire discipline as a CM technique, but this isn’t a requirements management journal.) CM tool vendors have a variety of requirements management solutions already available, and the biggest tool vendors—Rational (now IBM) and Telelogic (now IBM)—offer requirements management tools that integrate with their CM tools. (DOORS famously integrates with nearly everything.)

The last source of requirements challenges, and the most common, is frequency of change. When requirements changes (including new requirements) occur too fast, the development team can’t keep up. Symptoms include the reappearance of defects or deviations that have been fixed, excessive trouble testing the product, problem reports citing recently-updated requirements, and customer complaints about unresponsive or inadequate development. If the requirements change cycle is significantly faster than the development cycle, the list of pending changes can grow ridiculous. When this happens, development may respond by asking for an incredibly long requirements change and review cycle, typically tied to the development cycle. This is a symptom, and the solution is to speed up the release cycle, not slow down everything else. Consider an Agile development approach, in order to tighten up the feedback from development to customers.

The organizational techniques can be tremendously valuable when dealing with requirements challenges. A Change Control Board is an excellent source of detailed analysis when composed correctly. If you already have a CCB that you fear might be wrongly constructed, consider creating an additional one within the development group(s) with the right make-up. The ‘correct’ composition will depend on the type of requirements that are presenting the challenge. Functional requirements will need a functional CCB, procedural requirements need an organizational or managerial CCB. The benefit here is that each member of the board is looking at changes from the point of view of their own area of expertise. Parochialism is a benefit, since proposed requirements and proposed implementations can be checked in some detail.

System Architect does not offer the same diversity of interests that a CCB does. However, a single person with the entire system in his head provides a different value. The System Architect is going to have a strong grasp of how the different parts of the system interact. In combination with a construction technique like Component-based Development or Service-oriented Architecture, the architect can review and analyze a larger volume of system changes than a CCB. When rate of change is the primary source of challenge, System Architect is a good technique for dealing with technical requirements.

In fact, a System Architect is essentially a specialized Subject Matter Expert in the context of requirements challenges. While a System Architect offers quick, informed understanding of technical requirements changes, a Subject Matter Expert can be created or hired for any kind of requirements challenge. This makes Subject Matter Expert the organizational technique of choice for rate-of-change challenges. Unfortunately, it can take months for a SME to get up to speed on a technical project. The particular nuances of a long-standing project take time to learn. Obviously the actual time required will be different for each separate aspect of the project. A DBA may come along quickly for table & index performance issues, but then take months to comprehend all the triggers and stored procedures that are used. Keep in mind that Subject Matter Experts are not strictly technical. There are plenty of experts available for process, management, and business requirements. (This is a shameless plug: the author is one of those available experts!)

The codevelopment techniques are useful to address rate-of-flow challenges. Simultaneous Development is an obvious requisite. There are occasionally situations (usually involving a common file or table) where serialized development is put forward as a solution to rate-of-flow challenges. It is better to make a simple process change to deal with this—choose one official maintainer of the file or table in question and assign all changes to that item to the designated victim. You may have to create “child” requests or tasks for this, but it is simpler and more effective to give all the funny stuff to one expert.

Of the ceremonious techniques, Automated Enforcement of Standards and Gated Workflow can help with requirements challenges when they are used in conjunction with other techniques. The object with these techniques is to devise a process that will help with requirements challenges, and then automate it. Obviously, Automated Enforcement of Standards is a win if the standard is one of the requirements. But it can also be used to execute test cases, which ties in nicely with Fast Feedback on Changes. Similarly, Gated Workflow can be used to involve a SME or CCB in the process, requiring sign-off before changes enter development. Gated Workflow can also help ensure that regression testing or impact analysis are done after a change is delivered. Finally, Requirements Management is a strong technique to use with rate-of-flow challenges. If a requirement is just unfamiliar, or merely incredibly difficult to implement, then requirements management may not offer much benefit. When a large number of requirements are changing, or when they are constraining the response to a smaller set of changing requirements, Requirements Management is the way to go.

(This was published as part of “Dimensions of SCM Challenge #2: Standards and Interfaces & Requirements or Business Demands” in CM Journal, Vol. 6, No. 3, March 2008.)

For our purposes, there is not much difference between a standard and an interface. Either one can be nearly trivial to support, or can impose enormous costs on the project. Complying with an internal coding standard, an ISO language standard, or a database vendor API can be simplicity itself. Trying to conform to a certain process model, or interface with a complex file format or communications protocol can supply problems for years. Part of this stems from the amount of documentation available for the standard or interface. Abundant documentation usually correlates with a straightforward implementation, especially if the documentation comes from O’Reilly & Associates. When the standard or interface is a moving target you can generally anticipate problems, but anticipating the problems usually helps mitigate them.

Sometimes the interface is all but undocumented. Every significant release of a major web browser is accompanied by a flurry of Internet activity dedicated to researching and understanding all the bugs and quirks in the rendering engine, and a parallel search for ways to uniquely identify the new version from within CSS, within HTML, and using Javascript. If you work for a popular web site or a maker of web design technology, or if your organization specializes in web design, then keeping abreast of these updates is crucial to your business. (If you’ve never dealt with the joys of browser detection, ask a co-worker or friend who has. Once you get them started, the stream of bile and frustration that emerges will curl your hair.)

Working with an undocumented interface that is crucial to your product but totally outside your control is one extreme form of challenge. That particular challenge is usually short-lived, since everyone else in the web industry begins exploring and documenting the browser quirks at the same time. Another kind of challenge happens when you are implementing a standard without any specification of the required outcome. While an open specification works well for requirements, it doesn’t work when different teams interpret the same prose to mean different things in the design. If each side of an interface expects the other side to handle object initialization, for instance, then the first integration test will be short, loud, and exciting.

One key ‘gotcha’ to beware of when dealing with a technical standard or interface is the need to explore these standards very early in the process. Even when you are ‘assured’ of documentation or technical details, there are often subtle dependencies on the ‘usual’ values or the order of certain items. For instance, a generic “set parameter” interface may actually require that parameter A always be set before parameter B. If your project doesn’t control both sides of the interface, you should allocate time early on to mapping out the context and expectations.

Interface Management is a defined part of configuration management. Sadly, few software shops have much (if any) experience formally managing interfaces. There is a technique, Formal Interfaces & Standards, that uses the constraints and rigidity of interfaces as a strength rather than dealing with them as a challenge. This is a question of choice—the technique relies on using interfaces that you choose to put in place, rather than externally-imposed constraints. In fact, once you have agreed to use a standard, it doesn’t matter how it came to be. Having voluntarily agreed to the constraints may make the situation more palatable, but the concrete challenges will be the ones listed above.

The lifespan of a standard or interface is one of the most important factors in determining the inconvenience it will cause. A long-lasting standard or interface will generally be better documented, more flexible, and can be incorporated into the requirements, architecture, and design of the product. It is unlikely to change at all, but if it does change it will likely be a minor modification or update. A short-term interface or standard will generally be seen only at the design phase and after. It may be modified, updated, or it may disappear entirely between one release and the next.

For example, a “supported operating systems” list is a long term constraint, with updates as new OS versions are released, and a gradual phasing out of old versions. (Current-day Windows applications support surprisingly old versions of Windows, going as far back as 98 or ME.)

Another example: the decision to support a particular version or release of Oracle might qualify as either a long- or short-term constraint. If the Oracle software is being bundled with the product, or is expected to be installed on a dedicated server for the product, then the required Oracle version can be changed in lockstep with new product releases (usually with a small fudge-factor for upgrades). But if the Oracle installation is expected to be part of a customer’s existing infrastructure then the range of versions supported will be much wider, and testing will need to be much more thorough.

One last example: if an interface is completely internal to the product—a gateway between two modules, or a file format—it is likely to be entirely under control of the respective development groups, and so can be changed from version to version. In this case, the only limiting factor is the agreement of the various teams that use the interface. This is no small thing, since some of the other dimensions of challenge can rear their ugly heads at this point. Using Formal Interfaces & Standards is a good way to isolate development teams that are divided by geography, language, culture, or politics. But using the technique transfers part of the challenge directly from those dimensions to this one—a geography or culture challenge is transformed into a standards conformance challenge.

Using interfaces is a boolean proposition: either the system correctly uses the interface, or it does not. (Which is not to suggest that all interfaces are simple!) This is normally determined as part of system testing. Standards are less sanguine. Some standards, like programming language syntax, can be verified mechanically. Others have to be enforced or checked using a higher level process. In some cases, like with a process standard, the higher level process is necessary just to implement the standard, and so an even higher level process is required for checking. Checking for conformance can sometimes be implemented either manually or with an automated process. The choice between automation and manual effort may be a property of the standard, or it may be a cost-based or effort-based decision. Opting to spend thousands of man-hours a few at a time in a manual process over months or years instead of spending tens or hundreds of man-hours automating a process is a management decision—frequently a surprising one.

The decomposition-based construction techniques (Component-based Development, Service-oriented Architecture, and Software Assembly) are all potentially useful, provided that the decomposition aligns with the interfaces and that the interfaces are at roughly the same level as the granularity of the decompositions. It makes no sense to try to break up a complex library in order to align it with a special memory allocation package. When dealing with standards, rather than interfaces, decomposition won’t provide any direct benefits. In this case, Software Product Lines might be useful if the standard can be viewed as a feature—that is, if the standard is not an all-or-nothing proposition but instead constrains a subset of the product, or can be presented as an optional element. Many programs offer installation options to select one or more languages or character sets for the user interface or for program data. If conformance to an accessibility standard were an installation option, SPL would be a useful approach. Similarly, if the standard in question is a technical standard like support for a particular file format or graphics engine then SPL may be a good fit. The Microsoft Office™ suite installation process used to offer file format translators for a slew of competing products as optional packages. An organization converting from another word processor or spreadsheet package could install the readers for those file formats, while other customers could save the disk space.

Of the organizational techniques, System Architect has relatively little to offer. Using a Change Control Board can be an effective way to manage changes to an interface or standard, since the board will likely be small enough to work effectively together but diverse enough to consider all the likely impacts of a change. The makeup of the board will dictate its effectiveness dealing with interfaces or standards. A board that is represented along political/organizational lines (1 seat for QA, 1 seat for IT, 1 seat for Development, etc.) will handle process standards well, but will likely not do well for technical interfaces. Conversely, a board broken down along functional lines (1 seat for electrical, 1 seat for sensors, 1 seat for control systems) will do well managing technical interfaces but may not fully grasp the nuances of a process standard. Having a Subject Matter Expert can be an effective way to deal with a standard or interface, provided the SME is expert in the standard. When the interface or standard is key to the project, a SME is nearly always a good solution.

Technical standards and interfaces generally don’t interact with the work flow techniques. Process standards are another story. Project Baseline is likely to be the SCM technique envisioned by the authors, designers, and implementers of nearly all process standards. As a result, using Project Baseline is likely to minimize the problems encountered when first adopting a process standard. Problematically, not all projects can use Project Baseline, especially not with recent process standards like ITIL. Longacre Deployment Management™ bridges between the Project Baseline and ITIL styles, and can provide enough of an “adapter” to help complex projects conform to common, consistent standards.

The codevelopment techniques can be useful with technical standards (as opposed to process standards) and with most interfaces. Independent Development Paths created at the right level can permit a group of developers to make changes to an interface without blowing up the development environment for the rest of the team. Likewise, Distributed CM lets separate teams work separately. If the organizational structure is aligned with the interface structure, this will help keep both sides of the interface(s) separate, but bring their work together for integration in a controlled manner. Finally there is Formal Interfaces & Standards. Recommending this as a response to Interface challenges seems a bit of a bromide—“when life gives you interfaces, make interface-ade”—but it works. For technical standards, fighting one interface problem with another interface is a straightforward Facade pattern, or an Abstraction Layer.

Standards conformance is a ceremonious activitity, and all of the ceremonious techniques are potentially useful. Requirements Management can help with high-level technical standards, and with low- and mid-level process standards. (Low-level technical standards tend to disappear behind a single requirement, which isn’t very useful. And high-level process standards usually control the process of implementing the requirements—they are too high level for requirements to be useful.) Automated Enforcement of Standards is pretty much a no-brainer, provided the standards are amenable to being enforced automatically. Treating interfaces separately, and using a tool like IDL or rpcgen to manage them fits into this category. AES is not restricted only to process standards. Lastly, Gated Workflow can help with process standards and human-centric technical standards by forcing the standards verification to take place early in the change life cycle. As an example, certain software controlling the administration of drugs is required by the FDA to be peer reviewed whenever a change is made. Inserting this peer review sign-off into the change request work flow is an effective solution to the constraint.

(This was published as part of “Dimensions of SCM Challenge #2: Standards and Interfaces & Requirements or Business Demands” in CM Journal, Vol. 6, No. 3, March 2008.)

Part of managing software development is dealing with the challenges that arise. Delivering software requires overcoming the challenges, or at least mitigating the attendant risks during the development activity. Generally, organizations work with a constant level of challenge. When one challenge is overcome, the organization will take on a new challenge. For example, when a project releases software that overcomes a technical challenge, it might then schedule a new release with a challenging timeline, or re-open the technical challenge by choosing a new target platform. The next challenge is not “just like the last one, only more.” An organization that successfully implements geographically distributed development does not follow up by adding another development office. The organization may well open another office, but doing so is now a normal operation; in order to maintain the level of challenge, some other “impossible task” must be undertaken.

While problems may have unique characteristics, there are a lot of common themes. This series discusses sixteen separate dimensions of challenge. The situation your team faces can be analyzed with respect to the dimensions listed here, and some appropriate conclusions drawn. Each dimension is dissected, and some approaches for dealing with it are provided. Here is the complete list:

• Capability
• Culture / Ethnicity
• Efficiency
• Finances
• Geography
• Human Language
• Knowledge Retention
• Location of Responsibility
• Organizational Structure
• Requirements or Business Demands
• Scalability
• Schedule
• Standards & Interfaces
• Synchronicity
• Technical Complexity
• Technological Diversity

The case study is here, and the technical details article is here.

I’m eventually going to incorporate LDM info as pages here on Doing Better. But for now I’m frantically editing the articles to try to improve their look — CMJ’s article import scripts really aren’t very good at handling a complex document.