The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

XML for data store/restore

I want to use XML for storing/restoring data in my application. I was planning to save all the data in the file, and load it back from the file all at once. While the user is in the application manipulating the data, the file is not used.

I am wondering if this approach is the most optimal. Is it better to not read all the data at once into memory but read chunk by chunk as they are needed?

I think most RSS readers deal with lots of XML files for data storage/access. If you know how they work, or if you wrote something similar to mine, I'd like to hear your comments.

Tuesday, January 17, 2006

Read and write it in one-go unless you have some good reason not to.
S. Tanna
Tuesday, January 17, 2006
I support automatic persistence in a GUI framework that I write, but I don't use XML, I use YAML which is structured in a neater way becoming easier to edit it later if the need appears. So, I know a little bit how it works for me and I know that it's not perfect yet. What do I think that could be better about it?

* support separated files, one for each window, as well as  the shared data among all the application code. -- the reason for this is to create some redundancy in case a file becomes corrupt and it does not put in risk the whole preferences of the user.

How it works so far:

* during application loading, it loads the preferences and calls the method "load_preferences" for registered components that support it in the loaded windows, passing to it its loaded preference as a parameter, which could be "nil".

* when a window is about to be destroyed, it calls the method "save_preferences" for the registered components just like in the loading, but this time the string returned by the method is used for storing the preferences of the component.

* each window needs to have an unique name which is used for differentiation of their preferences; each window saves and loads its last position and size automatically if it supports persistence.

* there is a global way to access the preferences besides through the windows way (which support anonymous functions I think, though I haven't used it yet), so depending on the way that data is persisted, it can be accessed from other parts of the program.

* whenever a preference is set, the persistence file is recreated based on the current version in memory, so if a crash occurs the user won't lose the last set preferences. -- there can be a batch of set preferences which save the preferences only in the end.

Well, I'm 90% happy with it so far. Future enhancements (maybe):

* provide some kind of redundancy for the preferences, like several files or versioning of the preferences (utopia?).

* make it work as seamlessly as possible.

Still source of problems:

* by making things too automatic, sometimes useless data can slip by and get dormant in the preferences file and memory.

* by making things too automatic, I can persist things beyond mere fields (variables), so when structure of data changes, it becomes an issue when loading the old data on the new structures. -- fix: try to be as fine-grained as possible; :-) provide redundancy; make sure only useless crap is persisted, like sizes and positions of Windows and stuff like that -- serious data from the users need a database.

I hope that gives you some ideas to accomplish your goals.
Lost in the jungle
Tuesday, January 17, 2006
Not sure what language you're working in, but this might help or serve as an example.
Developer #13
Tuesday, January 17, 2006
Hey, did the YAML guy have to write a YAML parser? I'm curious.
Spinoza Send private email
Tuesday, January 17, 2006

It is a stream of a Hash using code like:

and, 'w'){|f| YAML.dump(@preferences, f) }
Lost in the jungle
Tuesday, January 17, 2006
Thanks, LitJ.
Spinoza Send private email
Tuesday, January 17, 2006
I use C++. It doesn't look like YAML has any C++ ports at this time. Shame.. :(
Tuesday, January 17, 2006
I wouldn't use XML unless someone is forcing me or paying me alot of money to. Couple of reasons:

1) XML is a meaningless term. XML is a metalanguage; everything really depends on your DTD/Schema, which turns it into a language. How well is your data model defined?

2) It's incredibly bloated.

3) YAML is much cleaner, lighter weight, and easier to read, especially if your data is more relational than hierarchical.
Steve Hirsch Send private email
Wednesday, January 18, 2006
>> I was planning to save all the data in the file, and load it back from the file all at once. <<

It all depends on how much data you're storing.  I would say that putting configuration info into an XML file is fine (as long as a normal human doesn't have to edit it).  But putting 100mb in an XML file is just asking for trouble.

Repeat after me: XML is not a database.
example Send private email
Friday, January 20, 2006
XML is a database; it is just a very very very bad one. I can't think of one metric (besides hype) on which it performs well.
Steve Hirsch Send private email
Monday, January 23, 2006
Check out:

Great Stuff for reading / writing XML in C++

XML is the future of data storage in files. Anything else is going to be run over by the huge XML truck whether you like it or not.

I figure it is better to get used to using it. XML works quite well, and there seems to be not much choice going forward anyway.
Warren Stevens Send private email
Monday, January 23, 2006
I think that you are probably right. It has lots and lots of weight behind it (kind of like Java did), and people will get it to "work" with lots and lots of hard work (i.e., time wasted from their lives). Can you point me to a place (outside of true documents) where it (or any other markup language language, for that matter) work well?
Steve Hirsch Send private email
Monday, January 23, 2006

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz