The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Application for uploading data

I am planning to start re-writing a program which seems quite generic and so might have already been done before. What I'm hoping to find out is whether anyone has experience doing this and can give me some advice on the design or point me to some useful resources.

(1) Import sales from each branch to a central server (currently, there are about 400 branches)

(1) Sales at the branches can be in different data sources (eg. SQL Server, Access, text files)
(2) The process starts with some event happening at the branch --- normally a file being created
(3) We will be using .net to write this application.

Potential Steps
(1) A windows service is installed at the branch with a FileSystemWatcher component
(2) The FileSystemWatcher watches for a file, which indicates that the sales are ready.
(3) When the file arrives, the service requests instructions from a web service (passing in a BranchID)
(4) The web service responds with a XML doc of instructions (based on the BranchID). These instructions would include things like: DataSourceType, SqlQuery, ConnectionString, TextFilePath, TextDelimiter.
(5) Using these instructions, the service gets the sales and transforms them into XML which is passed to the web service, and ultimately into the database on the server.

(1) Need to be able to change the instructions. eg. the FilePath of the text file datasource might change or the SQLQuery might get an extra criteria.
(2) Need to be able to add more instructions, so that we could upload more than just the sales from the branch in the future.

If anyone can offer me any advice on the design of such a product or point me to any useful resources, I would really appreciate it. Maybe something already exists and I'll be better using that.

Craig HB Send private email
Wednesday, October 04, 2006
Craig, do not write a line of code. What you are looking for is named Microsoft BizTalk and has been around for a while.
The entire product class is named EAI (Enterprise Application Integration) and the other superstar is TIBCO.
Sevenoaks Send private email
Wednesday, October 04, 2006
BizTalk is a fine product and all but don't expect to use it without "writing lines of code". The difference is that BizTalk and other competing products will offer to handle all of the communications stuff and provide integration points where you can insert code to perform the custom mapping and database queries. So they certainly can provide some decent benefits, especially if you  ever need to scale it out or add additional communications protocols. But you can expect the amount of work to be very similar given that you will need to spend considerable time on learning BizTalk or whatever product you choose.

Beware of products that claim to handle the whole process for you automatically without writing any code. It just isn't so.
I'll be the ass
Wednesday, October 04, 2006
And you may want to look into using Message Queuing technology. Web services are fine but they require a more complex error handling mechanism. For example, if that special file appears and the web service is down, what do you do then? You've "missed the opportunity" and the logic won't be retriggered again because the file is already there. In this case you would have to store off some indicator to go back and try again later. This makes the logic much more complex. Using message queues, you would simply write the message to a queue and forget about it.

Just be aware of the differences when designing your application. Many people underestimate the amount of code it takes to handle retries if a web service is down.
I'll be the ass
Wednesday, October 04, 2006
As someone who is in the process of creating a client/web-service combo in .NET for uploading industrial process data, I can vouch for the difficulty in handling errors. The only alternatives I've looked at are FTP and BITS. BITS seems a little extavagent, and FTP lacks in security. SFTP is not an option on my current shared server.

The only message queuing tech that I know about is email. Can anyone elaborate on the other ways to do it?
Josh Send private email
Wednesday, October 04, 2006
OK, so I did some research and found MSMQ. I am developing a web application on a shared server for now, and want to try messaging for uploading data instead of webservices. Is it worth my effort to use send and receive email with c# code versus upgrading to a dedicated server to have access to MSMQ?
Josh Send private email
Wednesday, October 04, 2006
You don't need a dedicated server to use MSMQ although it would probably be recommended. MSMQ comes free with Windows XP Pro. It isn't installed by default but you can easily install it by using Add/Remove Windows Components from the Add/Remove Programs applet in Control Panel. Win XP just has a simple client and has some restrictions on how you access the queue. But it is probably plenty for what you want to do.

Writing to and reading from a message queue is dirt simple. You can do it easily from C# using the System.Messenging set of classes. Look for examples on the web.
I'll be the ass
Wednesday, October 04, 2006
I forgot about MSMQ, whicH BT can handle. I used it few years ago for a crytical application and it worked very well.
Sevenoaks Send private email
Thursday, October 05, 2006
If you are interested in message-based solutions, you should invest a bit of time (and money) to read:

Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions
by Gregor Hohpe & Bobby Woolf

Hardcover: 686 pages - Addison-Wesley Professional 2003 - ISBN: 0321200683

Really reccomended, and fairly platform agnostic (examples are in Java/JMS, C#/MSMQ and TIBCO Rendezvous.
paolo marino Send private email
Friday, October 06, 2006

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz