The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Having trouble explaining benefits of continuous integration

I have been trouble explaining the benefits of implementing continuous integration for some of our projects to my boss and coworkers.

We have recently moved to TFS and I would like to get the team using MS TeamBuild.

We have three main types of projects:

Standalone Desktop applications

Small internal web applications

ETL / Data Warehouse processes using SSIS

While I think I have them convinced for the desktop applications, they seem to be having trouble seeing how the TeamBuild environment would be helpful for web applications and the SSIS package development.

My main arguments for using TeamBuild have been:

The build server ensures that the latest version of code checked in is compilable and passes the latest unit tests.

All of the external dependencies are known and checked in to SCM.

Ensuring a repeatable build process.

Can anyone help provide some other benefits of having a a CI process and a dedicated build server?

Thanks for any replies
BigJimSlade Send private email
Tuesday, October 28, 2008
 
 
"Can anyone help provide some other benefits"

You already mentioned the _main_ benefits. If they don't get them, just give more reasons and explain them in more detail, but don't try to find other "hidden" benefits.
Daniel_DL Send private email
Wednesday, October 29, 2008
 
 
Personally I'm a "continuous integration skeptic".

> The build server ensures that the latest version of code checked in is compilable and passes the latest unit tests.

Just because it is compilable and passes unit tests doesn't mean that it is of good quality. I'm not saying that it doesn't have some benefit. But to what degree is debatable.

> All of the external dependencies are known and checked in to SCM.

You don't need continuous integration for this but it can help.

> Ensuring a repeatable build process.

This one I can buy. But we have five developers on our current project and we all build individually. And yet it appears to be "repeatable" since we have never had any issues. This is probably due to having very strong version control processes that make sure all of the dependencies are in place.

Again, I'm not saying that it isn't beneficial to use continuous integration. But like all development processes it comes at a cost. And figuring out if the benefits outweigh the costs is the hard part. Every shop will have to evaluate the benefits/costs for their own situation. Perhaps your coworkers are looking at the costs vs. benefits analysis closer than you are and feel that the extra expense isn't justified for the web app and SSIS projects. Have you tried talking to them directly about why they don't feel it would be beneficial?
dood mcdoogle
Wednesday, October 29, 2008
 
 
We're also using TFS, and we're not using TeamBuild because nobody wants to dedicate time to learning it & maintaining the jobs.  That's a cost the company is just not willing to bear at the moment.

Eventually, maybe.  In the meantime, our well-understood & easily maintained build scripts get run most nights.
a former big-fiver Send private email
Wednesday, October 29, 2008
 
 
ooops, hit Enter too soon.

Our main app has been Powerbuilder, so finding test tools that work well with it has been problematic.  Now that we're moving to .NET, we'll have the opportunity to imnplelment real, valid test scripts.  That might provide the extra push to use TeamBuild or some other tool & continuous integration but it's months off for us at this point.
a former big-fiver Send private email
Wednesday, October 29, 2008
 
 
@dood you make a good point. I think it is definitely a cost that was to be amortized over the lifetime of the project. because there is a lot of upfront effort that goes into making it work. I think that is one of their issues.
BigJimSlade Send private email
Wednesday, October 29, 2008
 
 
A large percentage of bugs in a typical project (I don't have the number handy, but I'm sure it's in McConnell's Code Complete) are found in the integration phase. If you hold off on integration until the end, you've got a potentially huge, unknown sized bug backlog you have to tackle.

A Continuous Integration system means that you DON'T hold off until the end. Instead, you integrate every time there's a checkin, and if there's an integration issue it'll pop right then and there. And if there's an issue you know exactly what caused it, because you know what checkin its in.

There's a paradoxical, but true, statement we say around here. "If it hurts, do it more often." Integration is definitely one of those things that hurts less the more you do it, as long as you have the automation.
Chris Tavares Send private email
Wednesday, October 29, 2008
 
 
(Disclaimer: I work for a vendor of a Continuous Integration product, so am naturally pro-CI).

The classic article by Martin Fowler describes things fairly well:

http://martinfowler.com/articles/continuousIntegration.html#BenefitsOfContinuousIntegration

It is really all about fast feedback.  We all know that bugs and other problems are introduced into the codebase over time.  Continuous integration can help you find many of them as soon as they are introduced, making it much easier to identify the change that introduced the bug, much quicker to fix it, and reducing the impact on the development team and customers.  Without CI, if someone checks in a bug it has no hope of being noticed until another developer hits it.  This can block that and other developers from working until the bug is fixed.  If there have been many changes since it was introduced, it takes time just to narrow down the person responsible.  This is all wasted effort that is easily avoided with CI.

Now of course CI will not catch all bugs, it is heavily dependent on how good your automated test suite is.  But it will certainly catch some bugs (even if it is only broken compiles) and the benefit of that is huge in terms of saving developers' time.

For the skeptics, the cost of CI doesn't have to be large at all.  If you have a scripted build, there are CI servers you can set up in minutes which will start bringing immediate benefits.  The more difficult cost/benefit analysis is around creating and maintaining a decent automated test suite, as this takes a lot more time.  In my experience almost all teams test too little rather than too much, though.  People see the upfront cost of developing tests right in their face, but can ignore the huge costs of extra support and lost sales that comes from shipping buggy software.
Jason Sankey Send private email
Thursday, October 30, 2008
 
 
Jason Said:

"Continuous integration can help you find many of them as soon as they are introduced, "

But only if the bugs are related to compiling errors. Some are, and most aren't. And I could argue that I don't need continuous integration to see these bugs. You will see them eventually. CI just means that you will HOPEFULLY see them sooner rather than later and the cost of fixing them is often dependant upon how soon you find them. But in my experience these types of bugs are usually very easy to fix regardless of when they are found. Compile time bugs that are difficult/costly to fix imply a breakdown of basic communication and design processes.

"Now of course CI will not catch all bugs, it is heavily dependent on how good your automated test suite is.  But it will certainly catch some bugs (even if it is only broken compiles)"

And that's really the crux of my argument against CI. Again, I'm not saying that it isn't beneficial. But there is nothing magical about CI that causes you to create better code. After all, you could abstract ANY manual checkin process as being "continuous integration". The only thing truly special about real CI is the time iteration. And that is really just semantics. For example, a programmer who always checks in code changes as soon as he is done is actually performing CI. Even if he only makes a code change once a year...
dood mcdoogle
Thursday, October 30, 2008
 
 
Here's the thing though - yes, you could manually check this stuff. But with a CI server, you have a machine that does it for you. On every checkin. That doesn't cut corners, get bored, or have to go to a meeting.

As an example, we just released our latest version of Entlib. About 6 weeks ago, we started seeing some odd intermittent failures on our CI server. At first we just thought it was a glitch, but it kept happening, but only on the CI server, not on our local machines. We dug in and found a subtle weak reference / multithreading / GC interaction that would, very, very occasionally, cause an object we expected to be live to get GC'ed anyway.

There is simply no way we would have found this if we were doing manual testing. And even if we'd seen it, the intermittent nature of the problem would have made it orders of magnitude more difficult to track down without the automatic runs of the build server, giving us more information on each failure. And more importantly, the fact that the intermittent failures went away is evidence that we did, in fact, fix the problem. And, of course, we added another test that caused the error to happen directly, so we know for a fact that it's fixed, and we also will know if it ever happens again.

There's also other things the CI server can do for you, too. For example, most devs do debug builds. When do you test the release version? How about on the CI server every time. Do you use static analysis tools? Day to day, probably not, since they slow down your compiles. But the build server doesn't care.

Is there some effort involved? Sure, you have to write tests. But you know what - that's a good thing! And over time the test suite will grow into a great indicator that your app works the way you want it to.
Chris Tavares Send private email
Thursday, October 30, 2008
 
 
ason Said:

"Continuous integration can help you find many of them as soon as they are introduced, "

But only if the bugs are related to compiling errors.

I'd rather be reminded sooner rather than later when I've done something that breaks compilation.

A good suite of tests will catch plenty of other bugs. As long as your tests are written properly you can safely modify the existing code without breaking existing functionality. That's more of an argument for the benefit of automated testing. CI takes this further by allowing several developers on a team to do this without having to worry so much about treading on each other's toes. Someone made the point that with good communication this isn't a problem but we've found CI actually improves this communication by identifying problem areas almost immediately so we can focus our discussion on the affected areas.

Automated testing and CI really do become more effective the more extensively you use them but even without a single automated test written yet there are some immediate benefits from CI that alone can be worth the upfront cost.
Luke CK
Thursday, October 30, 2008
 
 
>> Continuous integration can help you find many of them
>> as soon as they are introduced
>
> But only if the bugs are related to compiling errors.

Not at all, to really do CI you need automated tests.  These catch much deeper problems than just compile errors.

> Some are, and most aren't. And I could argue that I
> don't need continuous integration to see these bugs.
> You will see them eventually. CI just means that you
> will HOPEFULLY see them sooner rather than later and
> the cost of fixing them is often dependant upon how
> soon you find them. But in my experience these types
> of bugs are usually very easy to fix regardless of
> when they are found. Compile time bugs that are
> difficult/costly to fix imply a breakdown of basic
> communication and design processes.

Even a compile time failure has cost the wrong person time to fix and interrupted their flow.  And when people have these issues it leads to other problems like a reluctance to update frequently because updates bring issues.  More importantly, the point of CI is to test a lot more than compilation if you do it right.

>> Now of course CI will not catch all bugs, it is
>> heavily dependent on how good your automated test
>> suite is.  But it will certainly catch some bugs
>> (even if it is only broken compiles)"
>
> And that's really the crux of my argument against CI.
> Again, I'm not saying that it isn't beneficial. But
> there is nothing magical about CI that causes you to
> create better code.

I'm not claiming it is magical in any way.  Just that it has benefits which in my experience far outweigh the cost.  I have never worked on a project that didn't have some sort of automated build/test, and all my colleagues on these projects were in agreement that it was very valuable.

> After all, you could abstract ANY
> manual checkin process as being "continuous
> integration". The only thing truly special about real
> CI is the time iteration. And that is really just
> semantics. For example, a programmer who always checks
> in code changes as soon as he is done is actually
> performing CI. Even if he only makes a code change
> once a year...

I would not call that CI at all.  You don't need a CI server to do CI, but you do need at least to run a build and test in a clean environment (non-dev machine) on every checkin.  And people need to check in as regularly as possible for it to really be "continuous".

And although you don't need a server, I don't understand why software developers would want to do something manually when it can easily be automated.  As pointed out by another poster, an automatic process is more reliable, not to mention less effort over time.  And the more often it runs, the more information you get out of it.
Jason Sankey Send private email
Thursday, October 30, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz