The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

should I bother with REST, SOAP for internal web service?

I wrote a wing of an app that does some very intensive number crunching - the sort of thing that can slow a server to a crawl.  Rather than integrating it into the code base, my dev team is looking into running this code on another server, and calling it from the main app whenever necessary.

Because we'll have multiple boxes talking to each other, the notion of SOAP or REST came up.  However, we are not releasing this app to the public - and the code will be called in a very predictable, simple pattern.

I'm inclined to say that SOAP or even something simpler like REST is overkill here, because we aren't publishing some kind of API for the outside world.  Has anyone else done something like this?  If so, any suggestions on how to do this in a simple, lightweight way that doesn't bring in the complexity you'd need for a genuinely public web service?
gee bee
Monday, June 11, 2007
 
 
REST is not that complicated, if you ignore all the jargon, it's about simple requests and simple responses....

what you do is setup your script to accept GET parameters, and return results in xml (or html, or plaintext if you like) and then setup apache's mod_rewrite to get nice-looking URLs.

it's very simple.
Totally Agreeing
Monday, June 11, 2007
 
 
It depends on the tools you're using.  If you're using .NET or Java, it's pretty damn easy to put up a SOAP web service that can be used in a REST way (i.e. it supports HTTP GET and returns somewhat simple XML).

If you use HTTP GET with XML, you're using REST.  If you use HTTP GET without XML, you'll waste more time writing and maintaining your own parser for your custom protocol than you would save by skipping XML in the first place.

If you're using any kind of Kerberos or Windows Authentication, just use host the service with .NET and expose it as a WCF web service.  Yeah, there's a billion lines of complexity *somewhere* in there dealing with all the ws-security and ws-whatever standards, but the tools in .NET and Java (and probably many other languages) are now or will soon be to the point where it's sufficiently abstracted away that it's much less work than trying to do a half-assed job of it yourself.

Remember, you used to have to jump through hoops just to call a DLL too.  Now you don't even think about it.  SOAP vs. not-SOAP will be the same way in a few years.
Richard P
Monday, June 11, 2007
 
 
FTP?  Shared network drives?

I realize you're trying to protect your intellectual property but quite often the best solution does depend on what the actual data is and what you've got in place so far.

Sometime way back in my checkered past, I had to process a lot of satellite data; literally GBs of raw data would be dumped on me every few days like clockwork, but they were less picky about when they needed the final product back. (The scientists kept regular hours unlike the satellites.)

And the simplest solution was to have the program poll the source directory, look for new stuff, move it out of that directory, process it, write it out to the destination, then check the source again. If there was nothing there, it would sleep 15 minutes, then check again.

I mentioned this as a possible solution because it does not sound like the user expects to sit there waiting for the system to respond. Instead, you can do something like a "refresh" button where the user checks to see if his calculations are complete.
TheDavid
Monday, June 11, 2007
 
 
Err...  after reading the others' responses, keep in mind that I'm contemplating the kind of computing tasks that would tie up a true minicomputer for an hour.
TheDavid
Monday, June 11, 2007
 
 
If your need the raw speed, you can always go for plain socket programming. It's not really hard, except that you'll basically be implementing your own protocol, which is anything but easily extensible.
Jeroen Bouwens
Tuesday, June 12, 2007
 
 
REST...  SOAP is the Java of... er, I mean the COBOL of this decade.
KC Send private email
Tuesday, June 12, 2007
 
 
If you are using any kind of "platform" (.NET, Java) then SOAP is so trivially easy to implement that it is a no-brainer; and the performance "costs" will be negligible in comparison with the total processing time, so... do it!
Syd Send private email
Tuesday, June 12, 2007
 
 
Sounds more like a message queueing application.  Surely these aren't synchronous calls?  If they are it sounds like a job for any kind of basic RPC you have on hand.
B. Afraid
Tuesday, June 12, 2007
 
 
I concur with the messaging approach.  Have you looked at JMS (assuming this is a Java app)?  Microsoft has a message queuing technology as well although I am not familiar with it.
Eric Knipp Send private email
Tuesday, June 12, 2007
 
 
With .NET, at least, implementing web services is insanely easy. Really.
Jivlain Send private email
Tuesday, June 12, 2007
 
 
I'd probably implement the asynchronous capability at the server side accessed through a synchronous interface (REST, SOAP, or whatever).

Basically, have interface commands to start compute jobs on the server, monitor progress and check to see if the results are available. In fact, it wouldn't surprise me if there was already a standard way of doing this.
Arethuza
Tuesday, June 12, 2007
 
 
Thanks for the responses so far!

Just to confirm what a couple of people have asked - yes, this is a process that could take up to an hour to run (though a more typical scenario would be 30 seconds - 10 minutes).  Most of the activity takes place between the application and a database server to populate and run a model with 100K+ variables, and write the results to a database.

The traffic between the web server and the "math" server (not sure what to call it here) is pretty minor - a few parameters (nothing that wouldn't fit into a URL).  The respose is also lightweight - just some information about whether the application ran, what problems (if any) were encountered, and so forth.  So basically, the traffic is minor, but the web app that calls it will have to wait a variable amount of time for a response before knowing what to do next.

Yeah, there probably is a standard way of handling this - it seems like the sort of problem that would come up in almost any data-instensive, time consuming process that isn't specifically database-related.  I've googled and asked around a bit, but if anyone has some links to white papers or articles, I would greatly appreciate it.
gee bee
Tuesday, June 12, 2007
 
 
"So basically, the traffic is minor, but the web app that calls it will have to wait a variable amount of time for a response before knowing what to do next."

This is the problem. By their nature, web browsers don't lend themselves easily to this kind of behavior.

I apologize if gee bee didn't mean to word it this way, but since I'm making a point, I wanted to emphasize the use of the word "wait". As far as I know, there's no clean, robust industry standard for making web browsers wait passively and indefinitely for a response. Push technologies failed largely because the user could close the browser, jump to another web page, or even crash.

If you really need to have the spinning hourglass or progress bar going for a full hour or so, you're better off writing a desktop application that uses http and other protocals to transfer data. That would give you the desired control over the "wait" feature.

Back to the original subject, you can construct your web application to periodically poll the server and ask if a job is completed. SOAP works wonderfully for this. If the data is there, it gets it. If it's not there, it returns an error message from which you can assume it's not finished yet. I would discourage you however from trying to get SOAP to send a query and literally wait around until the data is finished.

In other words, decouple the "processor" from the monitor.
TheDavid
Tuesday, June 12, 2007
 
 
You aren't too far off the mark with your interpretation of "wait".  I mean it generically - the app doesn't have control over how long the process will take, and won't be the one running it.  Pinging the response periodically would certainly be one way to do it.  Or alternatively, I suppose the calling app could open a listener... anyway, I'm still looking around for solutions, so I'm trying not to phrase things in a way that leads to one approach over another.
gee bee
Tuesday, June 12, 2007
 
 
Hi Bee Gee

Maybe my response will be overkilling, but what you describe is something that is done every day in any scientific laboratory. The usual approach to solve is is using some kind of grid computing platform. If you are not familiar with the term, don' be afraid. It is basically an infrastructure to allow distributed execution of cpu and/or data intensive processes.

There are plenty of tools, but I would suggest you to use Globus Toolkit 4 (GT4 for short), because it has some nice built in features: an ready to use soap based interface to submit/query job execution, a nice Java API (Actually, it has a native one and another simplier one called CoG Kit) and also the ability to subscribe to changes in the state of a running job. The official site is: www.globus.org

The "only" tricky part is the security management, which is based on certificates, but it shouldn't be too complicated for a one shop environment (it is designed o be used in a multi shop environment)

Pablo
Pablo Chacin Send private email
Tuesday, June 12, 2007
 
 
If you're building a long-duration process that will be called from another process, the best way to manage that communication is asynchronously:  the client makes the request, and the server notifies it when the task completes.

You can implement asynchronous callbacks in a web service.  But I'd give some thought to using Message Queue as the transport here.  It's at least as easy as a web service to implement, and its asynchronous model is well implemented and robust.

Ultimately it doesn't really matter what method of transport you use, as long as it is asynchronous.  You can even just run the task in another thread (though that has the disadvantage of killing the task if the user exits the calling program).
Robert Rossney Send private email
Tuesday, June 12, 2007
 
 
Thanks again for the feedback.

I was a little worried that SOAP would be too heavyweight for what I'm doing, but it does look like there are some fairly easy ways to do this (with Java).

So far, Apache Axis looks like a promising as a way to go from Java to a web service very quickly.  I did the quick tour on their site, and it was exceptionally easy to get a simple app going from a server pov (not sure how difficult this will make things for the guy doing the client side of this app).

Anyone had any experience with Axis?  So far, it seems promising.

Wednesday, June 13, 2007
 
 
It seems your service takes an extremely long time, I think you need to think about:
- Is it run by many users? Recent PCs are very good, and the app could be run on their Desktops, for better performance (If many off them run it at the same time).
- It could be implemented in a clustered kind of way.
- Optimization of your code is very important, programs that could take up to an hour would benefit from any optimization.

What I would do in this case:
- Create a simple messaging service.
- Create Desktop apps can send requests and check for the status of a request, and get the results when finished.
- Create apps that would run on the desktops as well as servers (maybe multiple instances on the servers), these would fetch the messages and process them.
- It would be best if a user request is divided into a hundred messages, and then the results of all of them gathered.

Having a server+20 clients running this would enable up to 50 processes doing the processing, so a task that usually takes an hour could be finished in 5-10 minutes.

It all depends on your algorithm, and whether it can be parallelized.
Totally Agreeing
Thursday, June 14, 2007
 
 
You will want to design two messages something like this:

1) Post a request with the parameters you want to run, it returns a handle to the request to use in 2. On the server, this launches a process to do the ~hour long calculation.

2) Get status or result, specify handle, it returns either a progress report or the actual results. The calulating process stores progress in a data object that can be accessed by the handler/servlet that takes this call.

These messages are designed so that they essentially return immediately (though I know that if you use REST the underlying HTTP GET is asynchronous).
onanon
Thursday, June 14, 2007
 
 
Hi guys

I agree with most of the comments and suggestions about creating services and implementing some kind of messaging. Just consider that there are tools that do this, so the question here is if it makes sense to create a custom solution for something that can be solved by a middleware platform.

Some 10 years ago I used to do a lot of custom solutions for this kind of problems, basically because most of the middleware was either too complex and/or used propietary interfaces. Another important motivation was that I was willing to learn the intenals of middleware.

However, now I found this rather difficult to justify, mostly when modern middleware is available as open source and uses a lot of standards like SOAP, which isolates the application from the specific middlewre product your are using

By the way, I now work on developing middleware, so I'm a little biased here ;-)

Pablo
Pablo Chacin Send private email
Saturday, June 16, 2007
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz