* The Business of Software

A former community discussing the business of software, from the smallest shareware operation to Microsoft. A part of Joel on Software.

We're closed, folks!


» Business of Software FAQ
» The Business of Software Conference (held every fall, usually in Boston)
» Forum guidelines (Please read before posting!)


Andy Brice
Successful Software

Doug Nebeker ("Doug")

Jonathan Matthews
Creator of DeepTrawl, CloudTrawl, and LeapDoc

Nicholas Hebb
BreezeTree Software

Bob Walsh
host, Startup Success Podcast author of The Web Startup Success Guide and Micro-ISV: From Vision To Reality

Patrick McKenzie
Bingo Card Creator

Business Idea: Craigslist monitoring website

I want to be able to monitor craigslist for specific keywords. I want it to be web-based, super-easy signup, notify me of matches via Twitter, email, etc.

Monitize it by showing google ads for whatever they are searching on, or by including ads in the notification emails.

The only ones I can find are desktop applications, which is the wrong way to build this. My sister wants it to, she uses a mac, which is another reason not to build a desktop application.
Ted Graham Send private email
Friday, June 26, 2009
>The only ones I can find are desktop applications, which is the wrong way to build this.

I reviewed all the web based search engines:

Most use Google custom search.  Why?  Because Craigslist will ban your server's IP address as soon as you start making thousands of hits per day.  They will also ban your referrer string once you start sending them a lot of traffic.

To be clear, a moderately successful Craigslist monitoring service will make thousands of requests per hour.  The NotifyWire.com grid is already there.  We won't get banned because we're distributing all the monitoring hits throughout our grid of user's computers. 

On the plus side, unlike most installed software, you don't have to keep NotifyWire running for it to work.  Other nodes in the grid will monitor your searches while your computer is off. 

Free users must participate in the grid with a minimum utilization rate; premium users don't have to participate at all. 

Let me know if you have any questions.
ian drake Send private email
Friday, June 26, 2009
FYI, I tried doing something like this w/ Yahoo! Pipes.  Last I checked, CL blocks their RSS feed from Pipes, which sort of p*sses me off -- otherwise it's a five-minute job.
D. Lambert Send private email
Friday, June 26, 2009
You can try to use YQL http://developer.yahoo.com/yql/ and write a query like:
select * from rss where url='http://newyork.en.craigslist.org/hsw/index.rss'
enthusiast Send private email
Friday, June 26, 2009

Thanks for the explanation of why this is hard.  It is unfortunate that there not an easy-to-use solution for the casual user. 

Perhaps you should allow users to register one search query using the notifywire website that would freeload on your existing grid.  Once your users are hooked, explain why they need to start running the software to have more searches or features.

Ted Graham Send private email
Friday, June 26, 2009
Thanks Ted.  There are a couple options I'm looking into for the future. 

To be honest, right now I don't have the extra capacity to start giving even one search away for free to non-downloaders. 

The free users with the software installed create the extra capacity for the premium users.  If I remove the software install requirement for even just one search per person, one of two things will happen (or both). 

1) People who would have downloaded the software don't. 
2) People who will never download the software take the free ride and reduce the capacity of the grid for everyone else.

Right now there is a perfect balance between free and premium users.

I'm looking into Air and Silverlight...I'm not sure how much control over the HTTP request I can have in either of those.  I did find an Air app for searching Craigslist and it's user-agent and referrer give it away as non-browser which makes it too easy to be shut off by Craigslist.  Sadly many people start off like this, just to get shut down soon as they get a little taste of success.
ian drake Send private email
Friday, June 26, 2009
This falls into a risky category in that you could lose your business easily through the actions of others.

Suppose you could implement your business plan, that CL cooperated.


Then one day CL decides to implement this monitoring capability themselves for free.  You are out of business.
cal_programmer Send private email
Friday, June 26, 2009
That's true, but have you noticed that Craigslist hasn't changed much in the last 10+ years?

The same could be said for any software vendor out there, suddenly Google decides your market has attractive eyeballs and decides to make free software in your space. 

In any case, there are multiple problems with Craigslist that I plan to solve.  Search and monitoring are just the low hanging fruit.
ian drake Send private email
Saturday, June 27, 2009
What's wrong with just using the RSS feed that craigslist provides you in the search results? Surely with an RSS feed you can use a service to get whatever output you want (Email, SMS, etc).
ADoss Send private email
Saturday, June 27, 2009
We use the iMacros addon for that kind of monitoring (aka web scraping). It easy to use and  since our macros/scripts are "handmade", they are not and can not be blocked by Craigslist (and other similar websites that we need to scrape).
Simon C. Send private email
Saturday, June 27, 2009
ADoss - Yes, I'm sure you could do something like that, but what if you don't know what RSS is?  What if the RSS monitoring system you use only checks once every two hours?  If it's installed RSS software, what if you shut off your laptop?

There are a multitude of reasons I started notifywire.com and designed it as I did.  If you've ever sold something of value for short money on Craigslist, you'd see that you get responses to your ad within 5-10 minutes.  The best stuff goes fast. 

Checking every hour isn't good enough, checking every ten minutes will get your IP address in trouble.  There is a difficult problem to solve here for some people, but not every casual Craigslist user.
ian drake Send private email
Monday, June 29, 2009

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz