A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.
I write the Best Practices column for Software Test and Performance Magazine. I'm working on a piece due Tuesday, May 15 on best practices in defect tracking. I'm wondering if any here would be willing to weigh in -- on the record (i.e., quoted in the article) or off (i.e., "according to one 10-year defect tracking veteran...")
Thanks for considering it. Broadly speaking, I am curious about the idea that you can spend too much time tracking bugs. That, in the age of mass collaboration, open source, agile workflows, Wikis, etc., bugs (eventually, messily) take of themselves via self-organization. I know there's a thread on Joel on Software on this point: http://discuss.joelonsoftware.com/default.asp?joel.3.485094.37.
I'd love to hear from folks willing to weigh in and be quoted in print!
Sunday, May 13, 2007
I hope the fact that I'm the president of a software company that sells a defect management system doesn't preclude me from offering an objective opinion on this.
That article couldn't have been more ridiculous if Mr. Thorup had purposefully set out to write an article on "Why You Don't Need to Track Bugs" for The Onion. Note that Lars founded a software consulting company. I think I'll strike back with my counter article titled "Do It Yourself Software: Why You Shouldn't Waste Money on Consultants and Just Write that Flight Control Software Yourself".
The royal They say it's good to be provocative on your blog because then people will pay attention to something you write, even if it's to point out how wrong and ignorant that you've been. I think Thorup took that to heart when he set out to prove why managing your project using a "bug tracker" as he calls it, is a heavy handed process which is not worthwhile.
Let's look at his "argument"...
"Track systems are seductive"... in other words, once you start using them, you'll end up using them even more.
I don't even know where to start with this. Does this mean when lots of people end up using a piece of software for its intended purpose, we're supposed to consider this a bad thing? Call me crazy, but to me this implies that the software fills a need for them.
He says, "The large number of bug reports hinders a clear overview and prioritization." No, the large number of bug reports means the software has a large number of bugs. If you don't have a system to manage this, how in the world do you expect to have your software function in the real world? How are you supposed to remember all these errors? He says "Ideally, you should be able to start correcting the error as soon as you detect it." So you don't need to remember them, you can fix them as you find them, which means you better not have any customers because if they find defects in your software, you'll have to ask them to pop open their debugger and fix the code themselves.
Thorup relents by saying "note that having a system for handling customer requests is an entirely different matter". I see... so if it is a customer request then it's ok to track it, but defects we should just close our eyes and stick our fingers in our ears and pretend they don't exist until they go away. How should we track defects submitted by people who can't write code? What about QA? Or beta testers? How do you know what to work on first, what to push off to the next release? How do you get an idea for your schedule if you haven't estimated how long it will take you to fix all of your bugs? On a team of more than one, how do you know who is supposed to fix what?
The pièce de résistance starts with "what you really need is a piece of paper". Remember that bug system with the thousands of bugs? Ours has 300,000 since the inception of our company which span many different software projects. Any legitimate software company that knows about source control, defect tracking, build systems and other historically *proven* concepts of software development will have something similar. Am I supposed to believe that I should be writing these down on a piece of paper (which is unsearchable, unassignable, and uneditable)?
"To conclude: Bugs should not be tracked: avoid them or fix them instead." Avoid them? Seriously? Remember that scene in Star Wars when Obi-Wan gets past security with the Jedi mind trick: "These are not the droids you are looking for." Try using that on your customers the next time your app crashes. "Our application is still running and your document has not been lost. Move along."
Of course his rebuttal may be that he writes bug free code. Any sane programmer knows this is not possible without the kind of systems in place that they use for life and death critical software, and they sure as hell use a tracking system at those places. I guarantee it. (In five seconds, I found three errors on his website which I could have entered into his bug tracking system if he had one. Good thing for him... now he gets to just "avoid them".)
I have to cut him some slack though. He was probably using one of our competitor's products when he wrote this article.
Mr Thorup cannot be serious. Practical reasons why there is a need for defect and feature tracking system:
1. Having one makes the whole process of development and associated decision making transparent for all stakeholders involved. Harder to play politics, easier to get stuff done.
2. As Michael has already mentioned having a defect and feature tracking system makes time and cost estimation easier. One can do a cost benefit analysis based on facts, not opinions.
3. Written defect reports help eliminate ambiguity in communication. One can add screenshots, links, graphs etc to help communication. Written reports help managing knowledge and promote common source code ownership amongst developers.
4. As Michael has rightfully noted not every stakeholder involved in a project can fix a defect or is able to write code. Moreover, many defects need more than one stakeholder involvement to get fixed. Sure, a simple coding defect in most cases can be fixed by a developer on the spot, however analysis or design defects usually require several people involvement to get it right.
5. When software delivered under a contract formally recording defects might not only be a legal duty, but may help in many other very useful ways. Like avoiding feature disputes.
6. Feature and defect tracking is essentially a glue that holds all of the development team together (designers, developers, testers, support specialists, managers etc). How else are you supposed to tell a tester, what the defect was, which defects got fixed, how the defect relates to requirements and what was affected as a result of the fix.
And the list goes on and on.
From my development experience point of view, Mr Thorup at least right about the issues with defect prioritisation. If you allow your customers assign priorities, then too many issues end up being "urgent". On the other hand if you don't you tend to miss information on what your customers believe is really important.
Monday, May 14, 2007
I can't get to the original article (server time out) but I really wanted to address your idea about 'too much time tracking bugs'.
I think the nub of the issue is really too much time in the unproductive analysis of bug trends. In other words, analysis/paralysis. Too often I have seen bugs rolled up into scary metrics which have no chance of accurately reflecting the conditions in the Real World(TM).
Consider this, if last month I had one bug and this month I have two bugs, my bug count has gone up 100%! Now, if we know the total number of bugs (1 and 2), it's really not all that scary but often the reality is obscured by the reporting method.
I also think the philosophy of bugs is backwards. We seem to think it is a bad thing that we have discovered a bug. I think its a good thing because now, at least, I know its there and I can plan to address it. Its the bugs I don't know about that cause me nightmares.
Sure, it leads to a discussion about product quality (a discussion that always comes out *after* the 'hang the quality, just ship the product' speech) but, if handled correctly in conjunction with a sane point release strategy, leads to a high quality product in a manageable time frame.
Too often we attempt to pretend that our software is bug free. Often bonuses are tied to 'reducing bugs'. If the wrong metrics are applied to the measurement of 'reduced bugs', people will be disinclined to report them. (Don't report that bug, it'll kill our bonus this month!)
Of course, as pointed out in a Dilbert strip, rewarding people for reporting bugs is also a bad idea (Wally - "I'm gonna write me a Winnabego" [from memory]).
The answer, I think, is to embrace that bugs are a fact of writing software and to *plan* and *manage* for the fact that bugs will occur. In my experience, being able to respond to a customer bug with a process, periodic reports and a schedule of when it will be fully addressed goes much further than pretending to be surprised that a bug has occurred.
Maybe a big off topic but I hope this is germane to the discussion.
I guess I am in somewhat agreement with the original article, though I would propose that a stack of 3x5 cards is the best approach. The reason? Visibility.
Bug tracking databases suffer from two drawbacks. One, the data is only available to the select few who have an account and password. Two, the backlog is only visible when someone makes the effort to look.
Using 3x5 cards, especially if you expand the definition to include printed e-mails, allows virtually anyone to enter a bug. Using a physical means that is prominently display ensures the bug is actually dealt with. Have a growing list of new bugs that have not been vetted? This will be visible on a daily basis. Have a large backlog of unresolved bugs? Again, there will be a constant visible reminder.
Storing bugs in databases results in evergrowing numbers of bugs punctuated with periodic purges through administrative closes. It is not enough to say the problem is "bad" people not doing there job; we need to realize that the database approach does not provide the necessary level of feedback for the process to work.
Monday, May 14, 2007
"Bug tracking databases suffer from two drawbacks. One, the data is only available to the select few who have an account and password. Two, the backlog is only visible when someone makes the effort to look."
Come on, you're pulling my leg now just to get me riled up aren't you?
They do not suffer from these drawbacks.
First the data doesn't have to be visible to only those with an account and password. For example, I've submitted four bugs to MySQL about their database and ODBC drivers and I'm not some "privileged" user on their site. All our customers who email us can see their cases.
Additionally this isn't even a drawback! We don't want random people to be able to see our bug database... that's the point of having it password protected. The users who need to see it (i.e. the people in our company) have logins. Why would I need Joe Blow to be looking at my bug database? And to top it off, its even *worse* if you use 3x5 cards because now the person needs to PHYSICALLY be there to see your bugs. This is like telling people email is not as good as writing paper letters.
"the backlog is only visible when someone makes the effort to look."
Isn't that true for your stack of cards too? I mean sure it would be nice if we could all have our own personal bug fairies that would float on our shoulder and remind us how many bugs we have and how close we are to shipping... This is like saying you can only tell if the sun is shining if you make the effort to look out your window. So?
And if you're tracking system makes it hard to figure this out, then again, you must be using one of our competitor's products :)
I have mellowed a bit on my position after others pointed out aspects of Lars' argument that I missed the first time through.
If you distill his argument out from the provocative lunacy, it's that you should prioritize which bugs you're going to fix early, and that bug trackers encourage delaying prioritizing.
This phony conflict is a dumb argument on both sides: it comes back to the fact that a dysfunctional organization or individual can make any process dysfunctional.
You may quote complete sentences of mine verbatim.
I don't think "Bugs will take care of themselves through self-organization" is a true statement.
All the data sources you called out tend to be a little chaotic. Bugs themselves tend to be chaotic -- they don't self-organize, it's often not immediately evident what the cause of a bug is, and it can take some work even to cause the bug to reappear. It can even take some work to get a consensus whether a bug IS a bug or a feature.
In this environment of chaos, thinking that bugs will "self-organize" themselves out of existence is wishful thinking.
Monday, May 14, 2007
'I don't think "Bugs will take care of themselves through self-organization" is a true statement.'
I think it is a false statement.
'All the data sources you called out tend to be a little chaotic. Bugs themselves tend to be chaotic -- they don't self-organize, it's often not immediately evident what the cause of a bug is, and it can take some work even to cause the bug to reappear. It can even take some work to get a consensus whether a bug IS a bug or a feature.'
Bugs are sneaky and unco-operative.
I handle bugs on an ad hoc basis. The first thing I have to do is replicate the bug. Sometimes, this is easy. Sometimes, it is impossible. In the latter case, I end up rejecting the report. Occasionally, more data will come to light later, and I will be able to replicate the bug then. Some bugs never do get replicated.
At the risk of getting Michael all riled up, I'll weigh in as well. :)
There is one distinct advantage to the 3x5 index card defect tracking system: simplicity.
One thing I've witnessed in many (if not all) software based defect tracking systems that I've used is their tendency to eventually try to put everything AND the kitchen sink into it.
One very well known defect tracking system that I used recently (and which shall remain nameless) had a very configurable, very powerful workflow system built into it. The company I worked for at the time immediately wanted me to come up with rules about who could update which defects at what time based on their status, priority, estimate, age, whatever. Yikes. I just want people to be able to do the right thing.
I have been privileged enough to work on a project that could track its defects on 3x5 index cards and it was a very eye opening experience. People assume you need a complicated bug tracking system in order to create good software and while that might be true for many projects, I now know it's not a universal truth.
There are definitely things that a software based defect tracking system can do that the index cards have difficulty with. Distributed teams for one. But distributed teams are a challenge all their own with an increased risk from the get go. Basically, you're paying for the distributed team with the NEED for a software based defect tracking system. To me, it's not an advantage but a symptom of a sub-optimal development environment.
We're using Fogbugz at my latest job and it seems simple enough for me, though I might comment that 7 priority levels are still too many. I certainly don't mind using a software based defect tracking system but I do keep my eye out for signs that I might be able to get rid of it. I don't expect to be able to, but the indication I could would tell me that things were going very well indeed.
Please feel free to use or ignore any of this.
Monday, May 14, 2007
I don't know enough for anybody to quote me and I'm not very darn good at tracking bugs. But there are two things I know as well as I know anything:
1. data + organisation = information: If you've got a stack of index cards then that's all you've got. You don't know a darn thing until you read through them and put things in order, identify relationships and trends, etc. While some might be able to do a bang-up job of that using only their brains, I need to keep track of where I am and what I've discovered. Being a programmer, once I've discovered that there are patterns, I write up some code to automate the processing. I don't care whether it's bug tracking or accounts receivable, a database is a wonderful thing.
2. If I did a better job managing my bug reports, I'd be producing better software and improving my chops at the same time. As it is, I do way too much 'poking with a stick'. I'm trying. I really am.
Monday, May 14, 2007
You need a bug tracker. End of story. The idea that you can "avoid" or fix them as the appear is not feasible in most real world scenarios. In addition identifying priorities and assigning prioritisation is a vital aspect of any project and can't be achieved without recording them in the first place.
However your choice of bug tracker is open for debate. For example, if you have thousands of customers then FB is a good choice, but in a small shop Excel can work just as well. Ditto for highly distributed workforce vs the corner of one room. Ditto if you have a project manager vs a lead developer (you have to give PM's something to play around with all day).
Tuesday, May 15, 2007
This topic is archived. No further replies will be accepted.Other recent topics
Powered by FogBugz