Building Ideals

One of the failing points of the classic Waterfall Software Development Process is the separation of implementation and testing. Many processes since have tried to overcome the obvious limitations imposed: the easiest of these by far is Frequent Builds, and the most useful of these is Automated Tests. Bold claims!

Frequent Builds

We know building our code ensures that everything still compiles; it's like saying the sun makes light, or traffic today will crawl. But as our development effort grows, this seemingly obvious check becomes more important. It pays, then, to make sure the check is happening often, and more, to make sure it's happening right. Luckily, or providentially, we have more and more sophisticated auto-build tools, such as Ant, Nant, Visual Build, and so on, which take the pain of auto-building away. Combined with simple scheduling, Frequent Builds are almost trivial. Build once a week, once a day, once an hour. Take your pick.

Automated Tests

Automated Tests add a whole new level of power to us. Every time the build runs, run your tests; then for every test you write, that's one case you never have to worry about again. We won't talk now about Zen and the art of writing automated tests, we'll pretend we all already know when and how to do that rigorously. When we talk about building, we should be including our automated tests as well. If a build doesn't pass its test, it didn't build correctly.

What does it all mean?

But, even assuming everything we've said to this point is perfectly accurate and true, there are still simple questions left unanswered. What's the best way to build? How often should we build? What does a build consist of, anyway?

First thing's last!

Builds should include everything needed for the program to run. This including source code (of course), libraries, image files, and anything else needed while the program runs. This does not include extra documentation, readme files, or pictures of Chichi, the dancing skeleton.

Frequency of builds is a touchy subject. I have been ostracized by bug-eyed zealots for suggesting that builds could happen more than once a day. I have bemoaned weekly progress emails. I haven't heard of wars being fought over the subject, but I'm sure minor skirmishes have. When it comes down to it, you want your source tree to always successfully build; if processor time was nothing, we'd build whenever the source tree was updated (commit, checkin, whatever). Of course, we can't do that... yet... so we need to content ourselves with something somewhat less ambitious. For active development projects, building twice a day is probably sufficient. For any project under even mild development, builds should happen at least once a day; there's no reason for less.

Now, for the Great Act of Building, itself. What should this ritual consist of?

Start Fresh

We want this build, and all builds, to be a perfect representation of the products being built. We don't want old, mildewed code to spoil our pristine masterpiece. Therefore, we always need to ensure we're building from the latest, greatest stuff; make sure every old thing is cleaned up - everything compiled previously and everything previously retrieved from source control.

Trust and Accountability

Removing all previously built components is well understood. Mixed code components tend to make for weird errors. But remove everything previously retrieved from source control? It really depends on how much we trust our source control mechanisms. If we know that SourceSafe is always going to correctly synchronize our local build, we wouldn't worry about this. Too many things can go wrong - software bugs in source control, network failures, power failures. We need to trust that when we start building, everything we have is brand new, and nothing slipped through the wires. This way, if a build breaks in some way, we know it's not related to retrieving the source, just as we know it's not from building the source.

Avoid Conflicts

When we do get the source, we want to make sure everything we're getting is on the same footing. We don't want to start synchronizing and have someone else check in code in the middle, such that we have half of that person's updates. Ideally, of course, source control would deal with this problem, too. Again, we can't take the risk of it being wrong. Avoid these kinds of conflicts by labeling the entire source we're about to build with: use a label like "Auto-Build version 34 attempt January 6, 2003". Then perform a "Get by label" from source control. Other engineers can stick more into source control without us worrying.

Ensure Validity

Of course, we still have to build the system. A failed build tells us the system is not valid. Hopefully this doesn't happen very often. But here, also, is where we run our nice set of automated tests. The more complete the test set is, the more confidence we have that the system we build is good. Here's an important step, though: if even one test fails, we need to treat it as if the build failed in the first place. The system is not valid. This may seem extreme at first; we can worry about the 20% of the time when a fail doesn't matter once we're catching the 80% of the time when it does. At that point, our tests should be catching these boundary cases for us, and we can still expect 100% of tests passing. We want our code to be Ivy League material, after all.

Be Ready to Go

We want our build process to be as complete as possible - best case is that the build not only turns the source into a runable system, not only tests that runable system, but finally packages the system into the distribution form. Java systems can get packaged into jars and even wrapped into the installation programs. Native code can be built into the executable for and zipped up with the necessary README file. We want to be able to grab a build from yesterday, or three weeks ago, and run it just as if it were the final release. Because it might be.

Store Result

Now that our build is done, we need to keep everything in a place we can find it again. Consider your source control for this; if we're Ready to Go with our installation files, we can check them in and let source control maintain the history. Be sure to use labels with this to avoid configuration issues. At the very least, we should keep a list of our builds available and named by date and build number. With the price of storage today, redundancy is more than worth the cost tradeoff.

Minimal Cost Builds

Ultimately, we want our builds to cost very little in terms of time and space. If two builds occur before source changes, we want to store only one result. When a build is complete, we want to clean up our unneeded build materials and the source. Why did we clean up first thing, then, if we're cleaning up last thing, too? In case something went wrong and we never got to the clean up step. If we're going to clean up first then, why bother to clean up at the end? Ideally we want a pristine system before and after the build; we really want our build to just get going, rather than taking the time to clean up after an old build. Hopefully everyone will clean up after himself.

The Process

So, what then is the ideal build process?

Step 1) Clean up the build space (Start Fresh)

Step 2) Update the version number (Avoid Conflicts)

Step 3) Label the build components (Avoid Conflicts)

Step 4) Do a fresh full "Get" from source control (Start Fresh, Trust and Accountability)

Step 5) Build (Ensure Validity)

Step 6) Test (Ensure Validity)

Step 7) Package (Be Ready to Go, Store Results)

Step 8) Clean up the build space (Minimal Cost Builds)