Archive for August, 2006
We cannot get started without making clear what the Agile Manifesto is. So let’s push that out quickly. Agile Manifesto is a document that states we should prefer:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
Being a curious person I tend to ask “why?”. Does following these principles make your process agile? If you want to be agile, you probably have to deploy these techniques, but are they enough?
It depends on how you define agile:
Brian Marick, Agile software development and Glade air freshener: “While I like the word “agile” as a token naming what we do, I was there when it was coined. It was not meant to be an essential definition. It was explicitly conceived of as a marketing term: to be evocative, to be less dismissible than “lightweight” (the previous common term).”
James Bach, Who Stole Agile?: “It was always my understanding that “agile” meant agile. The Agile Manifesto looks to me like an enumeration of the factors that allow software development to be agile. That’s why I like the manifesto.”
So if you define “agile” as the marketing form of lightweight, you’ve probably done enough when you’ve follow the Manifesto. But if you want agile in the meaning it’s defined in dictionary, that is, quick, the Manifesto can lead you the way, but not take you to the destination.
As marketing can assign some good-sounding label to anything and find blind followers, we’re not interested in the first definition. We’ll rather take a look why should we produce software fast. Why exactly is faster better? Why should we take agility further than states the Manifesto?
Karel Kravik: “There’s simple truth behind the scenes but nobody has yet stated it in that way.”
Software has time value
Time value is best explained in money example: €100 now is worth more than €100 in a year – if you had €100 now, you could put into deposit and earn interest during the year, that is, if the interest rate were 10%, you would have €110 at the end of the year. Yet there is another factor that eats away your interest earnings – inflation. Inflation is a property of money to lose value during time. So if during the same year inflation is 3%, you end up with €110 having the purchasing power only about €106.8.
Wikipedia thoroughly explains the time value of money.
Now picture you’re a custom software developer. Your clients order software from you because:
- They can cut more costs than they pay for your software
- They can earn more money with software than they pay for it
People don’t order software because they find it fashionable to use it instead of holding paper documents in archives; they lose money while not having it.
If you’re a software product developer, it’s even simpler – as long as you cannot sell your product, you’re burning cash developing it.
Basically, not having software costs money.
It may not be simple, but if we really wanted, we could calculate the expenses of Hansabank when all the payments now going through Internet bank were made in bank offices. We only had to take the number of payments and an average cost of payment which, in turn, depends on the average salary of cashiers in different regions…
Imre Lumiste, Changelogic technical lead, also engaged in job interviews: “That’s a good impossible question.”
Karel Kravik: “And here is a rhetoric question – have you ever asked yourself why we see so much beta software nowadays?”
We see beta products because they’re there to gain mindshare, to get publicity, to build community around the product. Now when you want to calculate future value of money, you have to know the interest rate, but can you predict future mindshare?
You don’t know the interest rate
Here the analogy ends; it’s not nearly as simple to calculate the future value of software, in fact, it’s impossible. What make it impossible are things you don’t know and cannot predict – you are not aware of the risks you’re taking nor the optimal criteria for setting your priorities.
Karel Kravik: “Imagine you put €10M into deposit and you don’t know the interest rate – would you prefer taking your interest yearly, monthly or weekly all other circumstances being the same? Or maybe even daily?”
The point I’m trying to make is that as long as you don’t have the software (or feature) you don’t know how it will affect the future value. Now it’s your choice if you want to figure it out once a year or once a week - the fundamental difference here is that the wider the time window is the bigger is the risk you’re exposing yourself to.
You’re exposing yourself to the possibility that maybe the customers don’t like your software; maybe it doesn’t satisfy their needs nor match their expectations. So they end up not buying it any more, in turn meaning no money for the developers. And isn’t it rational to find that out after a month rather than a year?
Here is the place where all the fluff about agility and adaptability starts to make sense – rather than trying to predict the future value long ahead, you check it periodically, the shorter the period the less you have to predict.
Karel Kravik: “Just like long development in isolated branches is more probable to result conflicts during merge, long development without reality check is likely to end up with conflicts between your opinion of what is good and your customers’ expectations.”
There is one more factor that comes into play when we’re calculating time value – inflation.
You don’t know the inflation rate
The other essential component when calculating future value of money is inflation – the rate by which money loses its value.
By default your software too loses value; it loses value even if it’s not ready. And it may even never get ready, because your competitor brings out a superior product half a year before you.
So when you’re planning a release you have to draw a line somewhere – feature A goes in, feature B doesn’t, can we afford refactoring or not; you have to draw the line what is “good enough”.
The problem is that “good enough” is perceived differently by developers and the people who really use the software. Developers like it if the code is elegant, abstracted to the right level, optimized to the right level, readable and maintainable to the right level. A programmer is actually never quite satisfied with the code; he can tinker with it forever.
Now from the standpoint of business people – they are satisfied if it looks beautiful and does what it should do, they don’t care exactly how ugly it might look to a developer.
The twist is that if you focus too much on, say, maintainability, it could happen that you will never have to maintain it because there will be no chance to sell it in the first place. Focusing on technical perfection does not increase your user base.
Karel Kravik: “It’s like taking a loan with 10% interest rate and put it to a deposit with 3% yearly interest rate.”
To turn that interest rate differential into your favor you have to be lightweight; you have to operate cheaply; you have to be really focused on the core concept and push it out as soon as you can; as soon it is “good enough”.
Karel Kravik: “Just as developers occasionally write temporary code to test technical concepts, software products or individual features can be considered as tests too – tests of business ideas. There is no point to design them for a long run until you are not sure they’ll survive.”
The Manifesto
While the Manifesto gives us some clues how to be more lightweight, that is, cheap and if development is cheaper, we can probably release faster too, it doesn’t explicitly state what is the reasoning behind this; neither does it tell us how to be quick.
Karel Kravik: “If I had to sum this post up in Manifesto-like manner it would look like this:”
- Working software now over working software in the future
- Frequent reality checks over long predictions
- Good enough over perfect
August 23rd, 2006
Villu Ruusmann, programmer, Changelogic: “Why are we committing these jar files to CVS? This is such a waste! We should keep them in Maven repository.”
Karel Kravik, product manager, Changelogic: “Why? How much costs this extra 200KB?”
Villu Ruusmann: “C’mon man, this just isn’t elegant solution! It needs serious refactoring.”
Karel Kravik, mumbling angrily: “Yeah, exactly. Elegance. Refactoring. J2EE, XML, BMW.”
What do we know that Villu doesn’t? And what does Villu know that we don’t?
Villu knows that setting up Maven repository and configuring your project to use it will take 8 hours.
What he doesn’t know is that 8 hours of his time will cost us, let’s roughly guess, €100. What he can calculate, but it doesn’t come to his mind, is that 200KB of disk space will cost as much as €0.00057 (assuming you can get 70GB for €200, probably even cheaper since I don’t follow the prices). Even if we store that 200KB file in CVS for 3000 times, it still costs us - only - €1.7.
So where exactly is the elegance in paying €100 to save €1.7?
Alternative cost
Poor Villu, we are still not finished with him. What can Villu do with 8 hours?
Imre Lumiste, senior programmer, Changelogic: “He could fix some bugs.”
Karel Kravik: “Oh really. What bugs do we have?”
The Answer I Would Like To Hear: “We have here bug number 3828, which is causing every 10th client turning down our software. Of course everybody knows that an average client will bring us €5000 a year. At the moment we have 10 potential clients per day, so we’re losing €5000 every day. The problem is identified; it takes 8 hours to fix.”
But what is the real answer? It goes something like this:
Imre Lumiste: “We have here bug number 3828. This is some strange memory leak in our transaction engine that is probably caused from miscommunications with the database driver so that every once in a while a transaction is timed out. We have to investigate it carefully and maybe even contact the vendor support.”
Mmm-mmm, what else can we do with €100?
John B., marketing director: “Hey man, this is our moment! I just got an offer to advertise our product on the first page of Arvutimaailm. And it’s only €99.”
Suppose I am not very technical person (that’s sad, but true) and the memory leak leaves me somewhat untouched as I didn’t get the Answer I Would Like To Hear, so I decide to spend my €100 on marketing and buy the advertisement in Arvutimaailm (Computerworld, Estonian PC Magazine).
What I don’t know yet, is that CEO of Krakozhia Programming Farms, Slobodan Skradin (older brother of Milan Skradin, CIO of Krakozhia Telecom, which outsources all development to Krakozhia Development Farms) will find this issue of Arvutimaailm on the plane seat next to him on his flight from London to New York and as the Changelogic commercial is the only dang thing in the language he can understand, that is, English, as the rest of the content being in some strange Eastern European dialect called Estonian, it grabs his attention.
The rest is history. After a month he signs an agreement to purchase no less than 3500 (it is what the name suggests - a farm!) Changelogic licenses along with another 650 grand worth of consulting, customizations and on-site training.
This is a single biggest software deal in Estonia after Skype founders got rich, which, in turn, doesn’t count because they sold their company, not their software.
Karel Kravik, in the cover story of Äripaev (Business Day, Estonian Wall Street Journal): “I’m telling my team every day that you must have clear priorities set.”
Nassim Taleb, Fooled By Randomness: “Lucky fool.”
We are option blind
Backspin.
Mister Krakozhia Programming Farms missed that plane; Arvutimaailm was never left on his neighbor seat because there was a small delay because of jammed printer so the plane left even earlier than first Arvutimaailms were carried out to subscribers. So it didn’t even reach the guy who could have left it in that plane. Nobody else looked at the commercial longer than it takes to turn the page.
Backspin.
You’re still in a point when you have to make the decision. You don’t know which scenario will become true, you can only guess.
Steve Jobs, Stay Hungry, Stay Foolish: “You can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future. You have to trust in something-your gut, destiny, life, karma, whatever.”
Nassim Taleb: “[This is what I call] Narrative Fallacy. Creating a story post-hoc so that an event will seem to have a cause.”
Problems with optimal choices
Firstly, people are probably not even aware of the options they can choose between. For example, I’m pretty sure, most of the technical staff will insist on fixing the bugs before continuing with new developments. Basically that is not bad, but there still has to be an economic justification. Some seasoned development leads choose between bugs and new features, but they never think that maybe they could use available resources in some area where they aren’t involved as day job, for example marketing.
Karel Kravik: “I have seen people fixing a ’serious flaw’ which consists of optimizing 10% something that takes 0.5% of overall request time and are happily unaware of whatever takes the rest 99,5%.”
Secondly, people are not aware of the optimization criteria; they cannot make the best decisions just because they don’t even come close to be aware of what the right optimization criteria might be. For techies it may be some kind of abstraction, optimization, refactoring, the Elegance of the solution.
Although, when your product ships, it’s very likely that nobody cares about the Elegance, except if the Elegance is the part of the solution; something included in your sales pitch. If you optimize a video game to run really fast, it’s justified, saving 200KB of disk space nowadays is not unless you’re optimizing Linux to run from 1.44′’ diskette.
Karel Kravik: “Sometimes this is due to poor communication, but during a lot of debates I’ve been holding, people simply do not accept the argument of economic justification.”
Raakel Kaakiv, Karel’s alter ego: “Maybe this is because the hype is blind, you either do refactor or don’t, 1 or 0. Actually there a full scale between them. What complicates the problem is that the scale is highly context dependent, what works in one project or team, won’t do in others.”
Rakuke Kakuke, Karel’s another alter ego, a better one: “Typically innovative things are not economically justified from day one.”
Raakel Kaakiv: “Would you be still in business after a year if all you do is innovation? Would the decision be simpler if you knew that your chances of success are 1:1 000 000 when you’re innovating and 1:100 if you spend your resources in a bit more risk averse way?”
Rakuke Kakuke: “Define what is more risk averse! I see no risk because I haven’t put a single penny in this business.”
Let’s leave these guys arguing, it’s usual, and see what happens if we knew the right optimization criteria.
Thirdly, people are not very good at evaluating the options they have. That is not because of these people were incompetent or plain fool, but because the outcomes are often unpredictable. Even worse, they represent the kind of unpredictability where you cannot calculate your odds.
And still, at the end of the day, you have to make choices.
Despair Inc, Indifference: “It takes 43 muscles to frown and 17 to smile, but it doesn’t take any to sit there with a dumb look on your face.”
Making it easy
Karel Kravik: “Just bringing out the key points this messy rant was about.”
So when you have to choose between multiple options ask yourself:
- Is this the full set of options? Do you see the whole picture?
- What are the actual criteria you’re evaluating your options against? Do you see the whole spectrum of criteria?
- Can you evaluate the alternatives? Can you predict the whole chain of events that will unfold after your decision?
Here is a little hint about the ultimate optimization criteria:
Joel Spolsky: “The real goal for software companies should be converting capital into software that works.”
August 14th, 2006
With this post I’ll open next top level topic - handling uncertainty in software development.
Let me list some risks from an average software development project:
- Will the functionality be useful for our customers? (”user risks”)
- Will there be showstopper bugs? Problems with updates? (quality risks)
- Will we hit the due date? Will we be in budget? (”resource risks”)
- (this is not full list, you can come up with your own items)
I’m lining them up in this order because they depend on each other in the same order. If users don’t find our product useful, our product cannot be considered as quality product and we must most certainly spend more money to make it better. On the other hand, if we have resource shortage, our product probably won’t be as good as we’d expect.
Now suppose you have exposed yourself to these risks and run into problems.
Jason Fried: “[You can create] Less Software. Less Software allows you to distribute your time and energy across less features. More attention to less stuff will make that less stuff better.”
Karel Kravik: “Sometimes cutting the scope is really an option. But it’s more usable in product development world and not always possible in custom software development where you have your functionality fixed in contract.”
Angry reader: “I didn’t know this kind of contracts still exist!”
Karel Kravik: “Sorry, they do. Go to your Department of Defense and say you’re now going agile. Gimme €15M and let’s see what we can do.”
The application management case
So when we’re starting to develop or maintain software, we’re always taking risks. Sometimes bigger, sometimes smaller but there is always some amount of it.
Milan Skradin, CIO, Krakozhia Telecom: “Downtime of my applications can not be more than 8 hours in a year under any circumstances. I’m losing thousands in an hour.”
Branko Slavic, project manager, Krakozhia Telecom: “I need to deliver new functionality at least every month.”
Yes, the point can be best made with software maintenance. Picture you signed SLA under these conditions.
Mihajlo D., release manager, Krakozhia Telecom, comments with an evil grin: “Keep in mind that every update takes minimum one hour. This is the absolute minimum.”
Although this post is turning out to be more like a play, we must give word to Kristjan Kanarik, project manager, Webmedia: “The only solution is that every update must be successful. We go through rigorous test procedures as usual, no, heck, we’ll double these efforts. We have well-documented acceptance and release procedures, we have excellent people. We can do it.”
Andrus Tamboom, perfectionist, Webmedia: “We’ll write a test robot.”
Karel Kravik, Changelogic salesman: “They use Changelogic to manage the process.”
Hubert, La Haine: “Heard about the guy who fell off a skyscraper? On his way down past each floor, he kept saying to reassure himself: So far so good… so far so good… so far so good. How you fall doesn’t matter. It’s how you land!”
Of course, you do everything to keep the risk down. Documented procedures, senior staff, doubled hardware. But it can only be a question of few minutes and you’re already breaching this SLA.
The problem is that you’re depending on some random outcome. We may argue that it’s not entirely random, yes, it’s just so complex that to a naked eye it’s indistinguishable from randomness. Suppose some anonymous (wished to stay anonymous) programmer made a mistake writing a shutdown procedure for the database engine you’re using. So the database shutdown takes 14 minutes, cold reboot, recover from backup, an hour and a half altogether instead of usual 2 minutes.
It has nothing to do with the professionalism of your team or the quality of your testing procedures. But it’s very hard to argue it was force majeure, too. Nor does it give you any means to justify the bad performance in your annual report.
You cannot observe the generator
No formal methods exist to manage this kind of risks; this is the kind of randomness where you don’t even know what your chances are.
Nassim Taleb, Fooled By Randomness, basing his example onto Russian roulette: “Unlike a well-defined precise game like Russian roulette, where the risks are visible to anyone capable of multiplying and dividing by six, one can not observe the barrel of reality. Very rarely the generator is visible to the naked eye.”
This is what he calls “unstructured randomness”, contrasting it to structured randomness like we see in, say, gambling, where you typically can calculate your odds.
Or to put it even more simply: we don’t know what we don’t know.
Probably the term “random” is what makes it really hard to accept, so let’s rather use the term “unpredictable”; it doesn’t change the result.
Karel Kravik: “Take any of the risks I mentioned in the beginning, call me amateur, but I say they are quite unpredictable. Users can change their mind, no software can be called bug free, estimates are rarely correct.”
My coworker, whispering: “I can almost hear the angry crowd.”
Karel Kravik: “What’s the problem? If they have accepted agile development for the right reasons, that is, because of better risk management and not because of they can write less documentation, they’re already accepting that software development is somewhat unpredictable.”
Martin Fowler, The New Methodology: “Predictability is a very desirable property. However if you believe you can be predictable when you can’t, it leads to situations where people build a plan early on, then don’t properly handle the situation where the plan falls apart. You see the plan and reality slowly drifting apart. For a long time you can pretend that the plan is still valid. But at some point the drift becomes too much and the plan falls apart.”
Martin Fowler: “Usually the fall is painful.”
Why waterfall doesn’t work
We had an example, quite extreme, from software maintenance, but let me bring another one from development world too.
Nassim Taleb, on ITConversations, author’s free transcript: “To predict the trajectory of one billiard ball - trivial, two - it’s OK, but to predict nine you need to take weight of every person in a room into account, because of gravity effect. […] To predict up to 53, you need to take into account every single particle in the universe.”
Karel Kravik, project manager: “Here is a list of about 400 tasks. We have to give estimates. This is a plan for our next year and a half in project Changelogic.”
Imre Lumiste, senior programmer: “Hmm, OK. Let’s say we implement the 10 with the highest impact first; should I be able to estimate how long will the next 10 after these take (and so on) considering they are depending on each other? Of course I can come up with some numbers, but they aint any better than your own estimates. Or any other guy’s from hallway.”
Karel Kravik, wannabee Dilbert character: “Be professional.”
It could be your project plan.
If nothing more, this is one single example of where your predictions are shaky at best. But if you use a bit of imagination, you’d probably find it more like an example why there is so much agility hype out there instead of highly predictive waterfall.
There will be no grande finale about the Changelogic guys inventing new risk management methodologies here, instead of, maybe next time you’re taking responsibilities, think how big may be the part you don’t know about; how surprising turns your venture might take because of the things you didn’t see coming.
Or at least how much you may be underestimating the error rate of your predictions.
August 7th, 2006
In a previous issue we discussed the evolution of branching models, we identified 4 levels of sophistication, but we didn’t solve the problems occurring on Level 2.
In this issue we take a look at how Changelogic branching model copes with these problems.
The problems when using the “branch-off on release” model were:
- Difference between mainline and release line may become significant, multiple merges bring even more complexity
- Customers cannot handle the situation when bug are fixed in production but not in latest test version
- Developers are not able nor willing match the risk associated with the change to right code line
Sergey, developer: “I committed the code and sent you the time report. My job is done. I’m pretty tired, I’ll go home now.”
Karel Kravik, release manager: “Where did you commit it?”
Sergey, looking angry: “Whaddaya mean where?? Into CVS!”
To find a relief to the first two points mentioned above, Changelogic deploys technique we call “continuous forward integration”. Don’t confuse it with “continuous integration” which is a lot broader term.
Continuous forward integration
Basically, if you would apply continuous forward integration to branch-off on release model, you get the situation where every bug fix committed to release line is also automatically merged into mainline. As a result mainline has every bug fix that is available in release lines from the moment they are committed and also may have some new functionality.
As the bug fixes are usually quite small pieces of code touching only a handful of files, the double integration does not need much effort. If there are conflicts, they are easier to solve compared to the situation where you have a lot of different bug fixes and you don’t exactly know which line of code belongs to which fix.
Sergey, the angry developer: “You mean that every time I commit something to release line, I have to merge it to mainline too? Come on, this is not optimal! I’m a programmer, not integrator.”
Karel Kravik: “There is a separate role called integrator, but you may also deploy the merge your own code pattern.”
Branko Slavic, project manager, Krakozhia Telecom: “At least if a bug is fixed now, it IS fixed unconditionally, I don’t have to mess around and look for the branches where it’s fixed and where it’s not. Or if it’s fixed at all.”
Karel Kravik, Changelogic salesman: “Actually there is a functionality called ‘version differences’ in Changelogic that enables you to find out what changes integrated between any two versions of your software.”
Early integration
Continuous bugfix propagation also promotes early integration as I understand it.
Karel Kravik, configuration manager: “I think the whole point of early integration is that the earlier you have a likely release package assembled, the earlier you can tackle the possible issues (compare it to the cherry picking philosophy, where you choose the changes shortly before release).”
Or to put it in another words: the earlier you discover problems, the cheaper they are to fix.
I don’t know where the problem is, but having spoken to many people about early integration, I found out in their understanding early integration means that they have no branches at all and they propagate committing (integrating) code as soon as there is any code.
Agile developer: “Have you read James Bach? He says: if it exists, I want to test it.”
Karel Kravik: “No problem, if you want to cooperate with tester, share your private environment or private branch.”
In a way I picture software development, changes with different risk levels cannot be integrated into one line, just because we misunderstand the early integration. I’m fully supporting early integration in the meaning I described - once you have established a release line, you should send your current most likely release candidate to testing, even knowing it’s really not there.
Which brings us to risk levels.
Risk levels (the tofu scale)
Laura Wingerd, Practical Perforce: “Release codelines are highest on the tofu scale; they are firm. They don’t change much, and even the slightest changes to them can impact release schedules because of their rigorous review and testing requirements. Development codelines are soft-they’re changing rapidly, the software in them is farthest from release, and there may not even be tests yet for their newest development.”
Karel Kravik: “Along with assessing the codeline’s ability to absorb risk, we should assess the risk of changes and match them.”
If we think about changes flowing into our application, we see that not all of them have the same impact to our application, if we generalize here a little, we’d get the following risk levels (one can always add, but this is a likely classification):
- experimental
- new functionality
- not critical bug fix
- critical bug fix
On the other hand we have release lines with various maturities:
- new development
- stabilization
- accumulated maintenance
- critical bug fix
All we should do is to recognize the risk level of change and match it to a release lines’ where it should go. It’s not one-to-one relation here and it may have some outside constraint like, say, customers wish “I want to see contracts list in the next week’s release”, but it’s also not so hard to map them. It would probably look like this:
- experimental change goes to new development line
- new functionality goes there too
- not critical bug fixes go to stabilization or accumulated maintenance line depending where the bug is found
- critical bug fixes go to stabilization or critical bug fix line, again, depending where the bug is found
There is one more detail we should pay attention - the release lines maturity changes over time, beginning with the new development and ending with closed, which is actually right after the “critical bug fixes only” level, with the difference that we don’t support it any more.
We’re practicing what we preach - let me bring you an example from project Changelogic itself (that is managed in Changelogic too). At this very moment we have the following developments:
- releases prior to 1.29 are not supported
- releases 1.29 and 1.33 get only very critical bug fixes
- release 1.36 is being actively maintained, including some minor new functionality
- release 2.1 is being stabilized, meaning it’s in production usage in-house
- release 2.2 is where the new functionality goes
- there are also some open changes containing experimental code
If you are looking for morale here, it could be something like this: differentiate between change risks and integrate them early to right release lines, and, if you like, try out the continuous forward integration in Changelogic.
Of course, this model does not come without cost; in the next essay we’ll cover parallelism, isolation and another agile practice “keep it releasable”.
August 2nd, 2006