The Joel on Software Discussion Group (CLOSED)

A place to discuss Joel on Software. Now closed.

This community works best when people use their real names. Please register for a free account.

Other Groups:
Joel on Software
Business of Software
Design of Software (CLOSED)
.NET Questions (CLOSED)
TechInterview.org
CityDesk
FogBugz
Fog Creek Copilot


The Old Forum


Your hosts:
Albert D. Kallal
Li-Fan Chen
Stephen Jones

Technical Debt is a fallacy

The thread "Measuring IT ROI on Projects" http://discuss.joelonsoftware.com/default.asp?joel.3.677317.5 touched on Technical Debt.  After looking at http://www.media-landscape.com/yapc/2006-06-26.AndyLester/  and http://www.martinfowler.com/bliki/TechnicalDebt.html and a few other discussions, I keep coming to the same conclusion, that these are smart people who understand why software development succeeds or fails, but they don't understand finance, and because of that, Technical Debt has no traction with managers on the financial side of the business.

Let's look at Andy Lester's analogy of having your house foreclosed on.  What a mortgage lender does is pay the full amount for the house on the day of possession and you have to pay back the lender over a period of time. If you don't pay the lender back, the lender gets your house.  You pay interest to the lender because the lender could invest his money elsewhere and the interest you pay has to be enough to convince the lender to buy a house for you instead of buying stocks or bonds or a house for someone else.

If Technical Debt really did exist, then the following scenario would have to exist:
A technology lending company would supply sufficient programming expertise up front for a software company to produce finished software.  Then the software company would pay for some of that expertise every month (principal), along with enough interest that the lender makes more money by supplying expertise than it does by selling its own software. As well, any future expenses related to supporting software are paid by the software company, not the lender.  This is similar to a home owner paying for utilities and maintenance, not the mortgage lender.  For this to make any sense, the software company needs to be certain that it can produce profitable software for as long as it takes to pay back the lender, but it is incapable of finding (or hiring) expertise that it can pay for on an ongoing basis.  Basically, in this scenario only the lender can afford to hire programmers (because they all need big guaranteed contracts?) but the lender can make more money by lending those programmers out than by producing software itself.  On top of which, if the software company defaults on its payments, the lender now owns the finished software product.

If the software company can't reliably predict future revenue from the software it sells, and can't be certain how much programming expertise is required to get that revenue, it has no way to calculate its own rate of return on investment with any accuracy.  Would you get a mortgage to buy a house that may not have enough room, could cost far more than planned to maintain, and might be worthless twenty years from now?  To make it worse, the mortgage lender is going to charge exorbitant interest because the likelihood of you defaulting is high, and the future value of the property is so uncertain.

When Cunningham and Lester and others talk about Technical Debt, they are really talking about the cost of bad management.  You could take out an insurance policy against the effects of bad management, but the premiums would cost more than what the policy would pay out (the insurer has to make money too.)  The supply of good (and bad) management isn't fixed, and the quality of management isn't random, so ownership is able to improve its management as long as it is willing to pay for it.  If ownership decides that the return on spending money to fix its management is too low, then it will discontinue the product, maybe even shut down completely.

The concept of Technical Debt assumes that bad management is inevitable, and it tries to build the cost of bad management into the cost of developing a product.  If a company makes decisions on new products based on a predetermined cost for bad management, it is handicapping itself, and is doomed to fail.
RGlasel Send private email
Thursday, September 11, 2008
 
 
"If Technical Debt really did exist, then the following scenario would have to exist:"

Is simply not true.
to lazy to log on
Thursday, September 11, 2008
 
 
You are carrying the debt analogy too far. Technical debt is just a cute way of referring to hidden "cost" of doing less than perfect, maintenance-free development. It appears to compound over time because all non-trivial projects have aspects where things are not perfect. The more of this you engage in, the more time you spend fixing bugs and the less time you have for new things.
Financial programmer
Thursday, September 11, 2008
 
 
Chinese firedrills are also a fallacy.

I've never seen one executed once where the car is on fire.
Bart Park Send private email
Thursday, September 11, 2008
 
 
RGlasel,

Companies do take out loans for technical and other activity all the time.  Ever heard of something called a "business loan"?

These sorts of things are discussed all the time in Managerial Accounting and Financial Accounting courses.  It's real.  It's nothing new.
Real World Survivor
Thursday, September 11, 2008
 
 
<<<Chinese firedrills are also a fallacy.

I've never seen one executed once where the car is on fire.>>>

Just because you haven't "SEEN" one doesn't mean it doesn't or hasn't existed in the past.
Brice Richard Send private email
Thursday, September 11, 2008
 
 
I think the OP makes some extraordinarily strong points in support of his position.

I hold and believe that software will NEVER EVER be able to be so predictable as to know how much it will cost to create software.

In its current incarnation, it just can't be done.

Joel claims that Event-Based Scheduling is the answer to software deliverables....my response to EBS is twofold: 1) it's micromanagement masquerading as some sort of software budgeting, and 2) trying to figure out how long software functionality will take in terms of min or hours is still just as unpredictable as trying to figure out how long it will take to build an entire app.

Everytime you develop an application you are embarking on a completely NEW journey....which is consquently why you can't predict with any reasonable degree of certainty how long it will take to write code in context to the application's EVER EVOLVING functionality.

Can't be done.....not today or yesterday.....something will have to change dramatically in the software development scheme of things to convince me that it's possible to do....
Brice Richard Send private email
Thursday, September 11, 2008
 
 
I stand corrected on EBS - the correct term is EVIDENCE-Based Scheduling not Event-Based Scheduling, for those of you who want to research the topic further.
Brice Richard Send private email
Thursday, September 11, 2008
 
 
I think that the ultimate cause of most scheduling problems is that for software development, the more you know what you want, the less you know how long it will take to develop and, the more you know how much time and money you have, the less you know how much you will produce.

It's a law of nature, but alas, one that doesn't fit into the current accounting game.
Steve Hirsch Send private email
Thursday, September 11, 2008
 
 
D. Lambert Send private email
Thursday, September 11, 2008
 
 
Consider a system that requires data import/export without providing an interface for non-technical users.  The worst case I've seen had a staff of 11 dedicated to recurring (not adhoc) ETL tasks.  The justification was that they were cheaper than developers.  Which was probably true at some point, but not when you have eleven full time employees.  Next year they'll need 12 or 13.
Defective Monk Send private email
Thursday, September 11, 2008
 
 
"Try McConnell's take on technical debt"

Oh god, not the stale old "what would McConnell do" reply.  Jesus, can you think for yourself?  McConnell's marketing and hype circus left town years ago!

I can't think of anyone more over hyped for spewing trivial and obvious advice than McConnell.
.
Thursday, September 11, 2008
 
 
"Oh god, not the stale old "what would McConnell do" reply."

LOL!  You beat me to it.  This circus ringleader's 15 minutes have been over with for a long time now.  He's a wordsmith at best.  The guy coded for what percent of his career?  Less than 1%?  His only claim to fame is that he pushed the Microsoft agenda by writing books for them.
TrixR4KidsSillyRabit
Thursday, September 11, 2008
 
 
@Financial programmer
The point that I used so many words to try to make, is that making decisions based on the concept of Technical Debt is the wrong approach.  There is no such thing as "perfect, maintenance-free development", so there is no golden standard to shoot for.  You can't determine what the "hidden cost of doing less than perfect" is, so don't decide to continue with or cancel a project because of what you think that hidden cost might be in the future. Instead, you should make those decisions based on the idea  that these are costs that can controlled over time, but not eliminated, and that the success or failure of your project will be determined by how well you control them. 

Debt is a bad analogy for these types of costs, because it implies that you should pay up front for future costs that are variable and controllable.  It is still possible to pay enough up front to cover all but the most unlikely costs in the future, but it is definitely a poor business practice.

@D. Lambert
"the obligation that a software organization incurs when it chooses a design or construction approach that's expedient in the short term but that increases complexity and is more costly in the long term"  That's the problem with Steve McConnell's piece.  He's trying to attribute an obligation (debt) to a business practice, not a financial transaction.  The only way to make it a financial transaction would be to force software developers to pay into a "bad software" fund, which would penalize good managers and reward the very same short-term, expedient management that everyone is complaining about.
RGlasel Send private email
Thursday, September 11, 2008
 
 
If you can't prove it wrong, claim that it's "trivial and obvious" and act as if that means it shouldn't have been said. Then people will think it must be wrong.

Also, never let a software developer near a soldering iron or a metaphor - they can't handle either safely.

Thursday, September 11, 2008
 
 
The "Debt" in this phrase has nothing to do with finance, and it has nothing to do with predicting future performance. It is the distilling of a whole bunch of imperfect variables into an imperfect assessment of the current state of affairs. It's akin to a bank pulling someone's FICO score in order to determine their current financial standing based on existing evidence and past behavior. Someone with a very low FICO score is in no shape to borrow money, in the same way that a project with high technical debt is in no shape to cut corners or take risks.
The Original Henry
Thursday, September 11, 2008
 
 
@RGlasel - "The only way to make it a financial transaction would be to force software developers to pay into a "bad software" fund, which would penalize good managers and reward the very same short-term, expedient management that everyone is complaining about."

I don't think that's the idea at all.  First, the idea of turning software practices into a financial transaction is to account for the difference between a system built well and a system full of short cuts.

A premise here is that a lot of managers think that if a well-build system (architecturally sound, documented, TDD - all the stuff you employ to build better systems) costs "X", then you can build the same system by short-cutting documentation, unit tests, good architecture, etc., for something less than "X" - say 0.8X.  I know that we'd all argue the truth of that premise, but most of us have worked for managers who believe it at some point.

The concept of technical debt says that the remaining 0.2X of cost doesn't just go away - it becomes debt.  You're going to pay for it later, when you support, maintain, and enhance the system, all of that work becomes more difficult because you didn't do the job right in the first place.  And, like financial debt, there's a carrying cost to technical debt that won't go away until you service the debt or dispose of the asset.

When the business decides to acquire software, they therefore decide to either fund the whole price (paying for proper construction) or to fund a portion of the software and take on a measure of technical debt equal to the costs they're cutting from development.

I understand that there are a lot of details that don't fit this model, not the least of which is the bit about speeding development by short-cutting good software practices.  I don't believe that anyone's suggested that technical debt can be tracked so precisely that it goes onto a balance sheet (though there are far more ridiculous things on balance sheets already).  This is a metaphor to help non-technical managers understand how technical decisions create business impacts.
D. Lambert Send private email
Thursday, September 11, 2008
 
 
In economics and finance, 'debt' can refer to all sorts of obligations, not just cash ones. There's nothing special about cash from the point of view of a lot of theory - it's just another tradeable commodity.
Matthew
Thursday, September 11, 2008
 
 
Also, not all debt is borrowed in one chunk (as with a mortgage), or repaid on the same terms as a mortgage.
Matthew
Thursday, September 11, 2008
 
 
One of the dangers of drawing analogies from unfamiliar topics to help illustrate them is that someone will nitpick at it and say "Oh yeah, well, here's a case where they're not at all alike," although it doesn't diminish the value of the core premise.  (cf. hand-wringing over "wiki gardening") So it is here.

I gave the talk at YAPC, and at many other conferences and user groups before and since, to help programmers and other techies learn a few important ideas:

1) That there is never a "later" to do the things that you should be doing now,

2) That pretending that there is a "later" is self-delusion,

3) That putting off important tasks is often, dare I use another financial expression, penny wise and dollar foolish

4) That the costs and benefits of programming decisions are ultimately expressed in time and dollars, because that's the language business speaks,

5) How to express those costs and benefits in dollars so as to be able to translate to your boss, impatient for a project completion, what the long-term impact of, say, not writing specs for the database engine will be.

If you don't like the term, that's fine.  I'd welcome another that's more fitting.  Until that comes along, however, since I've spoken with dozens of people who have told me that the term has helped them communicate with management much better and make more sound decisions on projects, I'll stick with "technical debt."
Andy Lester Send private email
Thursday, September 11, 2008
 
 
It doesn't work. Give any business person the opportunity to borrow an amount now that doesn't go on the books, that obviously can't all togeher be really quantified and that has an 'iffy' feel to actually ever having to be repayed, and they'll jump for joy and ask where they can get more.

Even if you somehow succeed in making the point that the repayment sometime in the future is real, the reaction will be the same.

As an engineer, you want to do it right. As a business person, you want to sell it right now.
Vee
Friday, September 12, 2008
 
 
I agree with Andy Lester in that Tech Debt is a /metaphor/.  Think of it like a parable in Biblical days - no, faith is not a mustard seed.  That's clearly not true.  Yet we can learn a lot from the story and comparison.

Really?  Do I really have to post that?

However, I disagree with Andy in that tech debt is always a bad thing, and the poster who said it's the result of bad management.

Too many technical people are wayyy too likely to assume the victim role.  "Oh, my manager wouldn't LET me do a good job.  It's his fault."

Hogwash.

When you (a developer) tell the manager it will take a week, and he says "That long?" you need to respond like this:

"YES, THAT LONG."

There are very, very few cases where you should take pressure from management to do a bad job for production work.

However, there are a few - I argue that the organization can and should take on tech debt when:

1) The programmers are masters who can actually measure and communicate the true cost of that tech debt,

2) The company needs to do _something_ now to make a big pile of cash very soon,

3) You have reason to believe that the debt will actually be paid.

Example:

A) A working demo to raise venture capital money (this is not production work),
B) A specific implementation for one customer who will pay enough for you to generalize and re-sell the software to other customers.

In August of 2008, I founded and led a peer workshop on the subject of tech debt, it was attended by three of the authors of the Agile Manifesto.  We video-taped the sessions and are currently working through the video editing.

Read about it here:

http://www.cs.calvin.edu/activities/technical-debt/

Regards,
Matthew Heusser Send private email
Friday, September 12, 2008
 
 
Technical debt is not appreciated by business or the OP for the same reason that it happens in the real world - intangibles and second-order effects:

Business is not good (despite claims about loans, finance, and insurance) at measuring relative value, especially when one side has too many intangibles or the effects are too indirect.

It's a fundamental problem of perception and limits of modelling in complex systems, and misreading curves by focusing on short-term effects.

The idea of technical debt is an expression of experiences about the expenses a system will incur indirectly or over long periods and views a system holistically.

If you want to see it in financial terms, you can actually see technical debt in action right now in the mortgage market meltdown, where short-term benefits were realized, without taking account the cost to be born later (by taxpayers and investors).  It's just shifting the cost.  It was always there, unacknowledged and unmodelled.
Cade Roux Send private email
Friday, September 12, 2008
 
 
Much like the original poster (or so I assume), I'm not sure that cutting corners on a project will always result in technical debt. The example I like to trot out in times like this, is of the sales manager that wants a one-time report based on data in spread out across multiple Access databases, by next Friday. You can design a true Visual Basic Application, or you can cut and paste into an Excel spreadsheet. The determining factor here is that management considers the application to have zero value after next Friday.

You, as a programmer may see value in the VBA approach since you'll be able to handle similar requests in the future a lot easier. From experience, it's a rare manager that can see into the future well enough to see that possibility.

I'm starting to think that there is something inherent in executive decision making that discourages from making that kind of strategic forecasting. Maybe its the concept of sunk costs and how it influences the decision whether to cancel or proceed with a project. Once the application is "delivered" and achieves the initial goals intended, it's paid for, it's over with, it's done.  Any subsequent maintenance is simply approved or rejected on the same terms as a brand new project, i.e., will spending X amount accomplish the goal?

If you're introducing the subject of technical debt into a discussion, you're really asking the manager to make two decisions. Namely, does spending A achieve goal B, and separately, does spending A+X make it cheaper to develop a hypothetical C that may never be needed (and if so, by how much)? Your experiences may be different, but I find it's hard enough as is, to get them to make the first decision, nevermind both.
TheDavid
Friday, September 12, 2008
 
 
"Everytime you develop an application you are embarking on a completely NEW journey...."

Well obviously that isn't true at all. There are only so many ways to use Access along with the clipboard. So that journey can't be new every time!
uggh
Friday, September 12, 2008
 
 
Maybe it's because I'm just not that smart, but I don't make complex calculations to determine the ROI on doing things "the right way". It's a gut feeling.

But I do believe in technical debt. Too many times, we've had a roadblock because of some legacy code that, if it had been designed properly, wouldn't be a problem. So we have to go back and spend time refactoring code that wasn't written the way that we would now write it.

Too many times we've fooled ourselves into thinking that some piece of code would be just temporary. In our business, that is rarely the case. It lives on and becomes a problem for new future development.

I know that today we are paying down our technical debt and it's costing us some time, but it's worth it. To each his own
Mark Hoffman Send private email
Friday, September 12, 2008
 
 
Once again, IT people thinking we're all unique and special flowers.

Call center rearranges the cubes to fit eight more people in. They don't have power or LAN drops where they need them. They can rip out the drop-cieling and start over, or they can splice like mad and have them in by Monday. But when you want *another* eight drops, you're screwed because you'd be overloading a circuit. That's technical debt.

There's a pothole on the bridge. You know that patching it means it will leak water into the superstructure, causing rust and premature failure. But patching it takes two hours, resurfacing the bridge takes two months. That's technical debt.

The fryolator has a busted alarm. Fixing it could cost a few hundred dollars. Or you can tell the fry guy to keep an eye on it and try not to burn too many fries. That's technical debt.


I'm not agreeing with the OP that technical debt doesn't exist. Just saying there's nothing at all unique about it. Choosing to let it build up doesn't make your boss an idiot.
Drew Kime Send private email
Friday, September 12, 2008
 
 
Wow, I didn't expect this thread to have such long legs, and I am impressed with the quality of responses.  I wish I could think of a better way to articulate this, but I think technical debt is more than just a casual metaphor, it is one expression of a mindset that believes it is possible to perfect software development (albeit with great effort). The metaphor of technical debt is used to justify the investment of resources to do a better job of developing software, not to improve the financial bottom line.  Software is seen as an end, not a means.

As far as I can tell, software development will never be perfected, and if it could, why would anyone spend the money necessary to get there, unless it was profitable to do so?  Personally, I would rather see the "invisible hand" of capitalism punish companies that cut too many corners, and if a blatant corner cutter doesn't get punished in the marketplace, well then perhaps corner cutting isn't always a bad thing.
RGlasel Send private email
Friday, September 12, 2008
 
 
What a storm in a tea cup?

The debt is real. What you cant count, may really count, Drucker?

e.g. I prototyped a product recently. Very happy and excited to create. Still I knew at that time that I was going "quick and dirty", that I was borrowing.

Then the debt turned out really high. I lost my enthusiasm once I realized I would take me more days to clean up than
to code it in the first place.

I must pay for the thrils and the highs during those "quick and dirty" sessions.

If it isnt debt, what is it? Please dont throw a theasurus at me. I am a "language" grad as you can see at http://ethicminds.blogspot.com/


If it isnt debt, what is it?
kyaw kyaw naing Send private email
Sunday, September 14, 2008
 
 
I've never said that "tech debt is always a bad thing," any more than getting a mortgage to buy a house is a bad thing.

The key is to understand that the costs of putting something off until later never decrease.  It will never take less time to document that database API, or write those tests.  The cost can only go up.

I make it very clear in my talks that I cannot answer the question of what debt is worth incurring, or what debts should get paid off first.  I just ask the questions that all of us as developers must ask ourselves.
Andy Lester Send private email
Monday, September 15, 2008
 
 
Andy -

For saying tech debt is not always a bad things, your presentations sure seem focused on getting rid of it.

I am pleased to here you speak of tech debt as a tool, like real debt, that can be used for ill or not. I've heard you speak on the subject, and watched your videos, and I can't recall you saying that before - but I don't have a perfect memory.

regards,
Matthew Heusser Send private email
Tuesday, September 16, 2008
 
 
"The key is to understand that the costs of putting something off until later never decrease.  It will never take less time to document that database API, or write those tests."

Not true. You've heard the saying that you should always expect to throw away the first version of anything you build? That means you throw out all the documentation of that first version also.

Okay, you don't really throw out the whole thing. But when do you document it? As you build? As you test? After release? The earlier you start, the more you will have to update your documentation to reflect changes that come up during development and testing.

Anything documentation you create that is not used before it needs to be changed is wasted money.
Drew Kime Send private email
Tuesday, September 16, 2008
 
 
"I am pleased to here you speak of tech debt as a tool, like real debt, that can be used for ill or not."

Well there are large numbers of people who would say debt is not a tool and one should never take on debt for any reason. Even though I don't fully agree with that, they do have valid arguments in their favor that should be considered when making such decisions.
The Original Henry
Tuesday, September 16, 2008
 
 
kyaw kyaw naing Send private email
Wednesday, September 17, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz