The Business of Software

So the CIO downloaded that giant dataset of raw search data, sorted by user id, that recently escaped from aol. He searched through it for all keywords relevant to our new product, and collated the results. They show pretty conclusively that there is absolutely no interest for the problem domain in our new product (under development for 1 yr), contrary to the marketing studies we did before. Big meeting today and I tried to argue that this is aol and it's only 20 million searches, but the fact that not one of those 20 million was relevant was a problem. So, the project I am overseeing has been cancelled and I've been told to select one or two employees to keep and the rest will be laid off on Friday. Fun times.

Art,

Does your target market overlap with the "People Magazine" demographic?

That may be unfair--I haven't looked at the dataset, so it could be that AOL users search for business and software-related terms, too.

Yeah, it was to be a consumer mass market product, sort of thing stocked in CompUSA and OfficeMax. So the aol demographics were relevant, but even I was shocked to find there was actually no interest whatsoever that could be even remotely construed as relevant.

I find it hard to fathom indeed that a decision like this could be made in this way. I've never used AOL in my life and could be exremelly interested in your product.

What market research was done in the first place? Were potential customers approached?

Sack the CEO I say.

What, your CIO has never heard of Wordtracker? They have _real_ search data.

Be careful, Art, that you too are not caught in the purge Friday.

Back-up anything critical to CD and get it out of the office today, just in case.

Couldn't he have gotten a rough idea of the popularity of the search by using Google AdWords, and either gauging the prices  asked or setting up some kind of dummy site?  Can you still do this in an attempt to save the product?

It seems outrageous to me that a project with a year-long investment already put into it would be scrapped so quickly over something like this.  Does this company have a history of making rash decisions?

Quick trigger fingers aside, is there not anything that can be done to save the project?  Maybe altering the main focus of the app slighty to point to hit a different (or more focused) target market or maybe changing some terminology or marketing?  This just seems like a big waste to me.

Highly recommend you read these comments by the AOL spokesman:

http://www.techcrunch.com/2006/08/07/aol-this-was-a-screw-up/

Bottom line, you are basing your decision on a very small subset of AOL users:

"According to comScore Media Metrix, the AOL search network had 42.7 million unique visitors in May, so the total data set covered roughly 1.5% of May search users."

Hope that's helpful.

Well, the demographics could well indeed just be an excuse. The main products have not been doing so well and this one is not ready to release, so killing it represents a way to save money by cutting costs. Of course the other products are at end of life cycle and have been neglected.

Yes, certainly after eliminating this project team, my head will be on the chopping block as I am overpaid already.

I'll bring my personal items home slowly this week just in case. Today I brought a stack of CDs out to my car and unfortunately ran into one of the developers, Tony, who was suspicious.

Tony: "Art, why are you taking all your CDs home?"

Me: "Eh, no reason ... Um, I'm just tired of these ones and will be swapping them for some others."

Tony: Pause. "So tomorrow you'll be bringing some more CDs? It'll be interesting to see if your tastes go beyond the Eagles."

Me: "Yeah, or I might bring them in next week."

Tony: "Next week? Why? Something happening on Friday I should know about."

Me: "Uh, no, um, why do you ask?"

I may have been a little too friendly here because Tony gave me a very strange look, and then said "OK. See you tomorrow." and then got into his car.

I can kind of understand the logic here - if out of 20 million searches, not a single one matched the problem being solved, then statistically speaking the odds weren't that good it would have shown up in substantial quantity within the other one billion searches (assuming that data set did represent 2% of all AOL searches).

However, for a mass market office supply store piece of software, I'm very skeptical that there really are no matches whatsoever - it's more probable that the CIO used the wrong search terms, or there was a bug in the SQL procedure or whatever. Were there any matches for common terms like "Jessica Simpson" or "stain remover"?

That said Art, while it sucks, maybe it was a sign that future decisions would be made with the same level of stupidity and pulling the plug now is the merciful thing to do.

Art, you sunnofabitch!!!!  Stale CDs my bloody arse.  Jerk.

Have you checked google trends?

Did the company did any research using Wordtracker/Overture keywords tool in the first place?

Good CIO.

The dataset does contain a massive demographic bias.

I would expect that technically savvy would not be using AOL's search feature.

For example,
- 94 people searched for 'xml'
- 1 person searched for 'xsd'
- 1 person searched for 'exif'

This does not mean there's no demand for these kinds of product.

Art, based on a similar experience I had a few years ago, I recommend you secure your good references, polish your own resume, and begin your search for a new job.

Keep in mind that these are the AOL users who do not even use Google for their searches, instead they use the AOL client to search.  I would be surprised if this demographic would ever even buy software at OfficeMax or CompUSA.  More likely their biggest software purchase would be those $5 poker game compilations at Wal-Mart.  (not making a value judgement about these folks, just an observation).

We don't have enough information to make the call whether the AOL dataset is or is not relevent to this company's product.

Assuming that is is, and that the CIO actually used a broad range of relevent keyword searches, then I too would be very concerned about getting no responses out of the dataset. That is a big warning flag that somebody is doing something wrong and it's time for a reassessment of the situation.

Canning the staff and cancelling the product seems like overkill, but as Art said, there are other issues at play as well.

Can anybody here help you find a new gig, Art?

Hey guys, thanks for all the feedback. Not too much time to post this evening, I'll try to give a big update maybe this weekend.

Some notes:

The company used a market research firm to look into viability. This involved paying people to respond to whether they think the product would be a good idea and would buy it. They also did a larger phone survey. The results were good here. I do not know if this sort of market research, which I think is the most common, has ever been actually tested to see if it is effective, or if so, what sorts of products it works with.

I was hired after the leads were poached by another firm. I've done a pretty good job, but I did take the best developers away from the maintenance teams of the legacy apps and put them on this project, which caused jealousy in some areas.

The other products this company makes are aged and date back to Windows 3.11 and it shows. Their sales have really fallen off, mainly because they are obsolete, and also there is freeware that competes with them and is better. So all the eggs are in the basket of the new product, unless there is something I am missing. The new decision seems to be suicide, but maybe the suicide was deciding to do this project in the first place, it's just taken a long time to realize that? But the fact seems to be that current revenue from existing products can't support further development and we are still several months away from deployment.

AOL is not all stupid people, it's regular, technically unsophisticated people. Not developers and not college engineering students, but everybody else in the country. Think of those applications you see that cost $50-$200 and are CAD programs for laying out your garden and fences and you'll know our demographic. (It's not CAD gardening though, but the project is still technically under nondisclosure so I'm not supposed to say much.) I do think the search information is relevant. I searched some of it myself late last night and showed that there were actually some terms you could find that would show interest, but it was no-go. Office supply store software has to be the sort of thing that will be expected to appeal at least modestly to 100% of shoppers, not 1%. You might not buy poker software, but if you had some you'd probably play with it. So-called 'long tail' products can't be marketed in big box retailers, so if just a small number are interested at all, it has to be sold some other way. The company needs big box level volume revenue to replace its previous revenue stream since this product won't ever sell for more than $99.

Anyway, I have open offers so I'll be ok when this blows up. Even if they don't lay me off, I'm going to bail. A lay off would just mean a nice bit of change to make the transition easier. This afternoon, I had lunch with the three top developers and let them know so they can leave. The two people I am going to 'recommend as the best' so as to not lay off are the two folks that are no good as developers and I am concerned about them because I think it would be very hard for them to find decent work in this field if they are laid off.

More later hopefully.

Art:

"all the eggs are in the basket of the new product...it was no-go...I'm going to bail...I am concerned about them."

Way to go!

For those who wonder what makes the difference between a Senior Developer and just a programmer... here it is.  A senior developer can consider the history of the company, the trends of the industry, the demographics of the marketplace, the business needs of the sales channel, the effect of management's decisions on the team.  These are things that transcend the choice of framework.  They need to be looked at first, before asking whether functions can be remapped in an particular language.

I guess the not-so-good developers will rearrange the deck chairs on the Titanic as it goes down.  Will you give them a copy of Yourdon's book "Death March?"  It might be a lovely parting gift.  Maybe the market research firm proves right.  In that case, they'll get to grow into lead roles from their seniority.