The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

How to store collections and hierarchies?

I want to store collections and hierarchies to a persistent medium and update them continuously.

In other words, I want to be able to have a collection that I can add items to or remove items from - both operations would be carried out a lot. I want to be able to persist each change to a disk, so in case of power failure or program crash, it can resume from the same point. It would be  a single long-running process which should be able to resume, if required. This collection wouldn't be a small one. There might be several collections distributed to several processes.

I also want to store several document in a hierarchical structure and tag properties to each document (and their containers). Again, I want to be able to store a huge no. of documents.

How should I store these collections and hierarchies? I looked into a few solutions, but I'm not so sure. Metakit and HDF5 might work for me. I'm not particularly looking for a RDBMS. Something agile and adequate should be more appropriate. It should also have a Python interface.
Wednesday, December 29, 2004
I don't know you you don't want an RDBMS.  Something like SQLlite should avoid most of the overhead you are talking about, but be lightning fast, handle all the disk i/o for you, an can be built into your code rather than needing a server running somewhere.
Joel Coehoorn
Wednesday, December 29, 2004
Isn't this exactly what ZODB (Zope Object Database) is for?
lw Send private email
Thursday, December 30, 2004
Double on the ZODB, you won't even notice you're storing them.
Simon Lucy Send private email
Tuesday, January 04, 2005

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz