* The Business of Software

A former community discussing the business of software, from the smallest shareware operation to Microsoft. A part of Joel on Software.

We're closed, folks!


» Business of Software FAQ
» The Business of Software Conference (held every fall, usually in Boston)
» Forum guidelines (Please read before posting!)


Andy Brice
Successful Software

Doug Nebeker ("Doug")

Jonathan Matthews
Creator of DeepTrawl, CloudTrawl, and LeapDoc

Nicholas Hebb
BreezeTree Software

Bob Walsh
host, Startup Success Podcast author of The Web Startup Success Guide and Micro-ISV: From Vision To Reality

Patrick McKenzie
Bingo Card Creator

SimpleDB, CouchDB and Other "NEW" Data Stores - Feedback

I've been hearing more and more about new types of Data Stores instead of relying on traditional databases. Can anyone share their experience with these RDBMS alternatives?  Am I missing the point or is it because I'm stuck in the mindset of small data needs?

I'm considering using Amazon's Cloud services for my next SAAS project but am concerned about lock-in with their SimpleDB offering.

Tuesday, January 20, 2009
For me the 3 most apparent problems are lock in, inability to debug/test locally and no possibility to have consistent backups. If Amazon provided a "developer" edition of SimpleDB that could be run on my computer I would think about creating my own backup solution and forgetting the lock in part.

To overcome the lock in concerns they could offer a version of SimpleDB engine that could be run on my own datacenter(s). I would probably never need it but psychologically the availability of such option could be all I need.
Not SimpleDB user
Tuesday, January 20, 2009
Forgot to add: There are lots of real-world usages which are fine with the constrains of SimpleDB. In fact there are many which are much better modeled with it than RDBMS.
Not SimpleDB user
Tuesday, January 20, 2009
I'm using SimpleDB to record Poker Copilot usage stats. It works great for that. I don't need to install and maintain a server. I don't need to worry about it falling over or failing to scale.

My scenario is write-lots-read-seldomly. If a piece of data gets lost here or there, I don't really care. I don't need things like foreign keys, data quality, and referential integrity for this usage.

Perhaps that's the sweet spot for SimpleDB usage.

I'm suffering though, from the lack of dev tools for SimpleDB. I've tried a couple but find it easier to write a few lines of Java code to get what I need.
Steve McLeod Send private email
Tuesday, January 20, 2009
Part of your question is architectural. Regardless of the choice of a database platform for an app, most developers are going to be somewhat tightly bound (i.e., locked in). A better choice would be to build an abstraction layer between your app and the code that accesses the database. If you do this, you can either allow the user to choose where to store the data (cloud or local db) or make you app easier to change if you later decide that you hate cloud databases.

Second, regarding backup/recovery, there are tools available for doing this. I am hesitant to say this, but my guess is that you are far more likely to actually lose data with a backup scheme that requires the user to keep the data backed up and being able to restore it if the need arises. Cloud databases are still in their infancy, but I think we can expect them to get more stable much more rapidly than we have seen with conventional databases systems.

Consider the overhead in keeping SQL Server or mySQL backed up, backups moved off-site, and the skills/time required to restore a dead database. It just doesn't make sense in lots of situations and I think many apps have chosen to store data in local files because there wasn't an inexpensive and easy-to-use option available. This is one place where I think cloud databases are going to be a tremendous help.

If you are start-up, consider the start-up costs (hardware, software licensing, personnel, etc.) associated with a data-intensive application. Cloud computing offers not only cost savings but that ability to scale to incredible volumes at near linear cost.

It doesn't really make sense to offer a developer version of a cloud database such as SimpleDB. Instead, you would simply create a test account on SimpleDB and test your app using the test account. Using tools such as SimpleDB Explorer (http://www.sdbexplorer.com - I am a happy user, not affiliated in any way), you can do all the normal things you can with most DBMS tools, making it easy to create and maintain test regions of your database.

In terms of functionality, SimpleDB uses a different paradigm than a typical 'relational' database. It took me a bit of time to adjust, but I have enjoyed the flexibility that SimpleDB provides over typical relational databases.
Michael S. Jones Send private email
Tuesday, January 20, 2009
I guess, like a post mentioned earlier, it depends on the nature of your data. We've been using CouchDB in production at wego.com as part of our hotels meta-search application for close to 9 months now.

At Wego, CouchDB is used to store details of each and every hotel that we have in our database. Since the 'details' of a particular hotel can be stored as a document with no need for 'joins' with other datasets, it is the right fit for our needs. For example, all the data on this page http://www.wego.com/hotels/singapore/singapore/pan-pacific-singapore is served off CouchDB (with a Rails-frontend)

We also have an internal billing application which is used to store partner invoices as CouchDB documents.

We were using the 'alpha' version of CouchDB (i.e. v0.7.3) until a couple of months back when we upgraded to the 0.8.1 version. The alpha version was able to handle our load quite well, but we re-architected our database and saw that as an opportunity to upgrade to the latest version.
Arun Thampi Send private email
Wednesday, January 21, 2009

Thanks for sharing your use of CouchDB in Wego.  Without divulging too much info, do you notice the larger size of the document database something of an issue as compared to use of a RDBMS as far as management and system administration?

Wednesday, January 21, 2009
Hmm yes, with the alpha versions, we were potentially running into some trouble with the database being too large to manage (i.e. backup, create new views and wait forever for the initial indexing to take place, etc).  But then during our re-architecture, we partitioned into smaller CouchDB databases which made it much easier to manage and backup.

From a total database size of over 40G, our largest partitioned database is now less than 100M (after compaction - another kickass feature), so the re-architecture + upgrading to 0.8.1 has worked well for us so far.
Arun Thampi Send private email
Wednesday, January 21, 2009
The main issues with simpledb are:
1. Learning curve. Many people are familiar with SQL, but few with SimpleDb. So if you need other people to work with you on the project, or you want to develop your own skills in this arena, SQL is much more attractive.
2. Much more primitive feature set. SQL is mature and has an immense array of features. SimpleDB is, well, simple and limited.
3. Lockin: what happens if you want to move away from Amazon? You're stuck.

The main advantage that SimpleDb has is that it's easy to deploy and scalable. But you're going to need a server anyway, so you might as well get a VPS or an Amazon instance and set up Mysql or Postgres on it.

Things might change in a couple of years as the product becomes more mature, but for now, that's my take, and I'm a big fan of S3 and EC2.
Dror Send private email
Wednesday, January 21, 2009

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz