Changelogic weblog

Evolution of branching models

This article intends to be an introduction to longer series about version control and branching in particular. On the way I also explain why things in Changelogic work like they work.

I’ll make a rough classification of project organizations here based on the sophistication of version control usage:

Level 0 - not using version control

Karel Kravik: “Honestly, I cannot imagine how any serious development can be done without it, but still, it happens.”

Level 1 - using version control, no branches

If we’re talking in CVS terms, all the development happens in MAIN and we only care about HEAD. MAIN and HEAD are CVS’s reserved tags for its internal bookkeeping, MAIN denoting the default branch and HEAD the very latest revision of any file we have in MAIN.

This is where almost everybody begins.

The first problem arises, when we’ve made our first release and are now proceeding with new development. As a rule, customers do find something they really really want to see fixed in that first release; usually there are plenty of these things. Now we face three options:

• we fix the bugs and hand over the release WITH the new alpha stage functionality we just started
• we let the customer live with the bugs until the alpha functionality is stabilized and do the release ASAP
• if it’s not the first release, it may be possible to roll back the last release to previous one, until new functionality stabilizes

Branko Slavic, project manager, Krakozhia Telecom: “Hey guys, don’t tell me it’s not possible. I want this bug fixed. Find a solution.”

Mihajlo D., release manager, Krakozhia Telecom: “COME ON YOU LAZY … (inaudible - ed.), FIX IT UP, I DON’T CARE HOW!!!”

None of the previous options is what could be considered as professional software development, as two latter ones leave the client unsatisfied and the first one drives us to a vicious circle where we find new bugs shortly after the release, fix them fast, release fast, just to find ourselves in the beginning again, because, in such a rush, we certainly introduce new bugs.

Taavi Kotka, Webmedia: “The more your software sucks, the more nights you need to fix it. You start with coffee, at some point it just doesn’t taste any more. Then you take caffeine pills, 6 pieces at 6 am keeps you going at least half of the next day. But still, I believe the ultimate solution is energy drink.”

Anton Masik, occasional writer: “I just love the chemical taste of it.”

Usual workaround: we introduce some kind of freeze, be it a code freeze or functionality freeze or whatever you name it, generally a stabilization period. It helps a bit, released software is of higher quality, for some time we can fix the bugs and support production, but sooner or later we’ve to start with new functionality. Meaning if production still needs support, we’re again choosing from the three bullets described above.

Freezes have another serious shortcoming - as stabilization rarely consumes all the resources and our developers end up sitting there doing nothing.

Level 2 - branch-off with release

This is perhaps the most broadly used solution as we can see it almost everywhere; maybe with little nuances, but the point remains the same. You can easily notice its popularity when looking at the open source projects with repositories online.

Branch-off with release means that every time we’re making a release, we create new branch right from the point in code history where the release has been made. Now we can proceed developing new functionality in MAIN and collect the bug fixes to the release branch. Somewhere in the future we merge the release branch with MAIN thus propagating the bug fixes into MAIN too.

Yes, we solve two of the hardest problems here - we can support production and we’re not wasting the resources during code freeze, but soon we still run into little and maybe not so little annoyances:

• as the release line evolves in isolation, the difference may become quite large and the merge far from trivial
• if you’ve shipped bug fixes to production and your customers sees test environment with pre-merge version, it creates a lot confusion and becomes one source of the “old bugs come back” problem
• developers cannot assess the risk of any change and don’t know into which branch it should go. The sad part is that they usually don’t even care.

There is actually one more problem ruining the picture - we might have to do multiple merges from release lines to MAIN, say, before branching off another release line. Doing so, we mess it up quite badly and if we want to still control the situation, we’ve to keep track of all the branches and merges manually, which is, as you already guess, not a pleasant thing to do.

Dave Jones, Buried Alive in Patches: “By the time Linus put out pre1, my tree was 6MB away from mainline.”

Karel Kravik: “It’s not exactly about using CVS, but rather analogous branching model. Interesting piece.”

Usual workaround: at this point people usually start dreaming about perfect version control that would keep track of everything, let them merge how they want avoiding dupes and resolving conflicts on the fly. In configuration managers’ jargon it’s usually called “cherry picking“, meaning you can add and subtract changes at will, especially before making release.

If it doesn’t work, let’s cut it. If the risk level is not appropriate, let’s merge it to another branch.

But this story has a twist - where will it lead if we can (re)package changes at will? It’s not so simple and perfect solution as it seems to be. Why?

Because there are always some hidden dependencies.

Subtract one change, test, you have to subtract some more, test, now you have to add one to make it at least compile and so forth. You cannot judge the application final state by just looking at the code, you have to test it. If you have more changes than you can count on your one hand’s fingers, you have a combinatory explosion in QA, exact opposite of common practice called early integration.

Albert Stone, mathematician: “Combinatory explosion happens when your search space grows at n factorial.”

Lieutenant Albert Stone, Marine Corps: “Picture this - line up a hundred soldiers in random manner and figure out if they are in order of height, if they aint, choose next random combination…”

Level 3 - anything more complicated

Basically anything more complicated is already a niche model, dedicated to solve problems like platform or locale specific code, distributed development and things like that.

Respectable work on branching patterns has been done by Brad Appleton et al.

In the next piece we’ll be looking at how some of these problems are handled in Changelogic, the configuration management software we’re developing.

Entry Filed under: Version Control