The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

File format for data export/import

My WinForms application runs with SQLite database and I am very satisfied with it. Now I need to add export/import feature to my application, so I need also data format for these export files.

First I thought that XML or just some binary format could work, but then I realized that I could use directly some small SQLite database file. The export database file could contain only a single table with export data, or it could even be a part of the main database with the same format.

Do you recommend to use directly a database file for exporting/importing of data? It would be probably easier for me to work with SQLite format, compared to another format. Users will not need to edit the export files in any way, and the data will be always imported only back to my application, possibly located on another machine.

Petr Veit
Sunday, August 10, 2008
I don't know anything about SQLite's format, but I'd question how often it changes. What if, for instance, your users had one system upgraded to the latest SQLite, and another not, would they still be able to move the files around? Can that case come up?

The advantage to your own file format is that you control it, and it can't go changing when you aren't looking. The disadvantage is, of course, that you are going to have to write some code.

The question is: Can you take the risk of a file-format change being imposed by a third party? If the file format did change, how bad would that be for your users and for you?

Basically, your decision needs to depend on the particulars of the situation. If you just use the file as a way to get data out of one database and into the next, and you know they will always be at the same level of all components, then who cares? If the files are going to stick around for a while, however, then you might be better served by an XML or some other format that you control.
Michael Kohne Send private email
Sunday, August 10, 2008
I'm sure I'm missing something but I understood that import/export was for moving data between apps, not machines.  Why not just compress and move the original file?
Brad Siemens Send private email
Sunday, August 10, 2008
If the data is very simple, see if CSV can work. A plain text file is very usefull because you can get it into a spreadsheet, do some simple scripting on it, import it into another database etc.
Tov Are Jacobsen Send private email
Sunday, August 10, 2008
What Tov Are said.

Plus, SQLlite might have a built in CSV unloader in it.  If not there are free CSV script file generators out there for downloading and use.

The same might be true for XML.
Walter Mitty Send private email
Monday, August 11, 2008
Thanks all for their suggestions.

I forgot to mention that the export files will be used to import data only back into my app. For example one user using my app will call another one to send him some portion of data, so he can import it to his copy of my app. I will cover Excel, SCV and other common exports separately.

SQLite format can change of course, but so far, the current engine can read all old formats, so this shouldn't be a problem.
Petr Veit
Monday, August 11, 2008
So this is more a synchronize than import/export?  You have the plumbing to open the files so creating a subset of the data in it's original format would minimize work as well as complexity, no?
Brad Siemens Send private email
Monday, August 11, 2008
"I forgot to mention that the export files will be used to import data only back into my app. "

Until you change your mind.
Walter Mitty Send private email
Monday, August 11, 2008
+1 to the flat file formats though I prefer to use the grave character, as opposed to the comma, but that's another thread. While your intentions are clear now, they might change, plus, since it is the lowest common denominator, you are protected against 3rd party changes that you can't control.

Usually, developing flat file import/export code isn't too difficult.
Steve Hirsch Send private email
Monday, August 11, 2008
If that data is basically a table, a flat file is the way to go. If you need more structure, XML has the advantage that there are libraries available, however the XML tags can take as much space as the data and the file size balloons. That annoys me a lot.

My real reason for posting is to caution that if you use csv files, sooner or later someone is going to skew that to an Excel file and that is big trouble as Excel has no compunction about changing the data, i.e. strings to numbers.
Nutmeg Programmer Send private email
Monday, August 11, 2008
Thank all for the suggestions, I will try to use some format that will be under my control then. I will start with a set of CSV files packed into one file and will see how it works.
Petr Veit
Monday, August 11, 2008
Be careful to make sure you properly escape your data that goes into csv.  You also might need to watch out for any potential for data values to change on conversion - for example, if you output a floating point value, and then re import it, you might not get the quite the same number the second time around.
Sunday, August 31, 2008
Be careful of "smart" conversion, too.  A nasty example:


Gene Wirchenko
Gene Wirchenko Send private email
Wednesday, September 03, 2008

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz