Changelogic weblog

Changelogic’s branching model

In a previous issue we discussed the evolution of branching models, we identified 4 levels of sophistication, but we didn’t solve the problems occurring on Level 2.

In this issue we take a look at how Changelogic branching model copes with these problems.

The problems when using the “branch-off on release” model were:

1. Difference between mainline and release line may become significant, multiple merges bring even more complexity
2. Customers cannot handle the situation when bug are fixed in production but not in latest test version
3. Developers are not able nor willing match the risk associated with the change to right code line

Sergey, developer: “I committed the code and sent you the time report. My job is done. I’m pretty tired, I’ll go home now.”

Karel Kravik, release manager: “Where did you commit it?”

Sergey, looking angry: “Whaddaya mean where?? Into CVS!”

To find a relief to the first two points mentioned above, Changelogic deploys technique we call “continuous forward integration”. Don’t confuse it with “continuous integration” which is a lot broader term. 

Continuous forward integration

Basically, if you would apply continuous forward integration to branch-off on release model, you get the situation where every bug fix committed to release line is also automatically merged into mainline. As a result mainline has every bug fix that is available in release lines from the moment they are committed and also may have some new functionality.

As the bug fixes are usually quite small pieces of code touching only a handful of files, the double integration does not need much effort. If there are conflicts, they are easier to solve compared to the situation where you have a lot of different bug fixes and you don’t exactly know which line of code belongs to which fix.

Sergey, the angry developer: “You mean that every time I commit something to release line, I have to merge it to mainline too? Come on, this is not optimal! I’m a programmer, not integrator.”

Karel Kravik: “There is a separate role called integrator, but you may also deploy the merge your own code pattern.”

Branko Slavic, project manager, Krakozhia Telecom: “At least if a bug is fixed now, it IS fixed unconditionally, I don’t have to mess around and look for the branches where it’s fixed and where it’s not. Or if it’s fixed at all.”

Karel Kravik, Changelogic salesman: “Actually there is a functionality called ‘version differences’ in Changelogic that enables you to find out what changes integrated between any two versions of your software.”

Early integration

Continuous bugfix propagation also promotes early integration as I understand it.

Karel Kravik, configuration manager: “I think the whole point of early integration is that the earlier you have a likely release package assembled, the earlier you can tackle the possible issues (compare it to the cherry picking philosophy, where you choose the changes shortly before release).”

Or to put it in another words: the earlier you discover problems, the cheaper they are to fix.

I don’t know where the problem is, but having spoken to many people about early integration, I found out in their understanding early integration means that they have no branches at all and they propagate committing (integrating) code as soon as there is any code.

Agile developer: “Have you read James Bach? He says: if it exists, I want to test it.”

Karel Kravik: “No problem, if you want to cooperate with tester, share your private environment or private branch.”

In a way I picture software development, changes with different risk levels cannot be integrated into one line, just because we misunderstand the early integration. I’m fully supporting early integration in the meaning I described - once you have established a release line, you should send your current most likely release candidate to testing, even knowing it’s really not there.

Which brings us to risk levels.

Risk levels (the tofu scale)

Laura Wingerd, Practical Perforce: “Release codelines are highest on the tofu scale; they are firm. They don’t change much, and even the slightest changes to them can impact release schedules because of their rigorous review and testing requirements. Development codelines are soft-they’re changing rapidly, the software in them is farthest from release, and there may not even be tests yet for their newest development.”

Karel Kravik: “Along with assessing the codeline’s ability to absorb risk, we should assess the risk of changes and match them.”

If we think about changes flowing into our application, we see that not all of them have the same impact to our application, if we generalize here a little, we’d get the following risk levels (one can always add, but this is a likely classification):

• experimental
• new functionality
• not critical bug fix
• critical bug fix

On the other hand we have release lines with various maturities:

• new development
• stabilization
• accumulated maintenance
• critical bug fix

All we should do is to recognize the risk level of change and match it to a release lines’ where it should go. It’s not one-to-one relation here and it may have some outside constraint like, say, customers wish “I want to see contracts list in the next week’s release”, but it’s also not so hard to map them. It would probably look like this:

• experimental change goes to new development line
• new functionality goes there too
• not critical bug fixes go to stabilization or accumulated maintenance line depending where the bug is found
• critical bug fixes go to stabilization or critical bug fix line, again, depending where the bug is found

There is one more detail we should pay attention - the release lines maturity changes over time, beginning with the new development and ending with closed, which is actually right after the “critical bug fixes only” level, with the difference that we don’t support it any more.

We’re practicing what we preach - let me bring you an example from project Changelogic itself (that is managed in Changelogic too). At this very moment we have the following developments:

• releases prior to 1.29 are not supported
• releases 1.29 and 1.33 get only very critical bug fixes
• release 1.36 is being actively maintained, including some minor new functionality
• release 2.1 is being stabilized, meaning it’s in production usage in-house
• release 2.2 is where the new functionality goes
• there are also some open changes containing experimental code

If you are looking for morale here, it could be something like this: differentiate between change risks and integrate them early to right release lines, and, if you like, try out the continuous forward integration in Changelogic.

Of course, this model does not come without cost; in the next essay we’ll cover parallelism, isolation and another agile practice “keep it releasable”.

Add comment August 2nd, 2006

Evolution of branching models

This article intends to be an introduction to longer series about version control and branching in particular. On the way I also explain why things in Changelogic work like they work.

I’ll make a rough classification of project organizations here based on the sophistication of version control usage:

Level 0 - not using version control

Karel Kravik: “Honestly, I cannot imagine how any serious development can be done without it, but still, it happens.”

Level 1 - using version control, no branches

If we’re talking in CVS terms, all the development happens in MAIN and we only care about HEAD. MAIN and HEAD are CVS’s reserved tags for its internal bookkeeping, MAIN denoting the default branch and HEAD the very latest revision of any file we have in MAIN.

This is where almost everybody begins.

The first problem arises, when we’ve made our first release and are now proceeding with new development. As a rule, customers do find something they really really want to see fixed in that first release; usually there are plenty of these things. Now we face three options:

• we fix the bugs and hand over the release WITH the new alpha stage functionality we just started
• we let the customer live with the bugs until the alpha functionality is stabilized and do the release ASAP
• if it’s not the first release, it may be possible to roll back the last release to previous one, until new functionality stabilizes

Branko Slavic, project manager, Krakozhia Telecom: “Hey guys, don’t tell me it’s not possible. I want this bug fixed. Find a solution.”

Mihajlo D., release manager, Krakozhia Telecom: “COME ON YOU LAZY … (inaudible - ed.), FIX IT UP, I DON’T CARE HOW!!!”

None of the previous options is what could be considered as professional software development, as two latter ones leave the client unsatisfied and the first one drives us to a vicious circle where we find new bugs shortly after the release, fix them fast, release fast, just to find ourselves in the beginning again, because, in such a rush, we certainly introduce new bugs.

Taavi Kotka, Webmedia: “The more your software sucks, the more nights you need to fix it. You start with coffee, at some point it just doesn’t taste any more. Then you take caffeine pills, 6 pieces at 6 am keeps you going at least half of the next day. But still, I believe the ultimate solution is energy drink.”

Anton Masik, occasional writer: “I just love the chemical taste of it.”

Usual workaround: we introduce some kind of freeze, be it a code freeze or functionality freeze or whatever you name it, generally a stabilization period. It helps a bit, released software is of higher quality, for some time we can fix the bugs and support production, but sooner or later we’ve to start with new functionality. Meaning if production still needs support, we’re again choosing from the three bullets described above.

Freezes have another serious shortcoming - as stabilization rarely consumes all the resources and our developers end up sitting there doing nothing.

Level 2 - branch-off with release

This is perhaps the most broadly used solution as we can see it almost everywhere; maybe with little nuances, but the point remains the same. You can easily notice its popularity when looking at the open source projects with repositories online.

Branch-off with release means that every time we’re making a release, we create new branch right from the point in code history where the release has been made. Now we can proceed developing new functionality in MAIN and collect the bug fixes to the release branch. Somewhere in the future we merge the release branch with MAIN thus propagating the bug fixes into MAIN too.

Yes, we solve two of the hardest problems here - we can support production and we’re not wasting the resources during code freeze, but soon we still run into little and maybe not so little annoyances:

• as the release line evolves in isolation, the difference may become quite large and the merge far from trivial
• if you’ve shipped bug fixes to production and your customers sees test environment with pre-merge version, it creates a lot confusion and becomes one source of the “old bugs come back” problem
• developers cannot assess the risk of any change and don’t know into which branch it should go. The sad part is that they usually don’t even care.

There is actually one more problem ruining the picture - we might have to do multiple merges from release lines to MAIN, say, before branching off another release line. Doing so, we mess it up quite badly and if we want to still control the situation, we’ve to keep track of all the branches and merges manually, which is, as you already guess, not a pleasant thing to do.

Dave Jones, Buried Alive in Patches: “By the time Linus put out pre1, my tree was 6MB away from mainline.”

Karel Kravik: “It’s not exactly about using CVS, but rather analogous branching model. Interesting piece.”

Usual workaround: at this point people usually start dreaming about perfect version control that would keep track of everything, let them merge how they want avoiding dupes and resolving conflicts on the fly. In configuration managers’ jargon it’s usually called “cherry picking“, meaning you can add and subtract changes at will, especially before making release.

If it doesn’t work, let’s cut it. If the risk level is not appropriate, let’s merge it to another branch.

But this story has a twist - where will it lead if we can (re)package changes at will? It’s not so simple and perfect solution as it seems to be. Why?

Because there are always some hidden dependencies.

Subtract one change, test, you have to subtract some more, test, now you have to add one to make it at least compile and so forth. You cannot judge the application final state by just looking at the code, you have to test it. If you have more changes than you can count on your one hand’s fingers, you have a combinatory explosion in QA, exact opposite of common practice called early integration.

Albert Stone, mathematician: “Combinatory explosion happens when your search space grows at n factorial.”

Lieutenant Albert Stone, Marine Corps: “Picture this - line up a hundred soldiers in random manner and figure out if they are in order of height, if they aint, choose next random combination…”

Level 3 - anything more complicated

Basically anything more complicated is already a niche model, dedicated to solve problems like platform or locale specific code, distributed development and things like that.

Respectable work on branching patterns has been done by Brad Appleton et al.

In the next piece we’ll be looking at how some of these problems are handled in Changelogic, the configuration management software we’re developing.

Add comment July 31st, 2006