The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

DCOM, RPC, raw sockets, or...?

Hi,

I am designing a COM server that has the following role: it provides "glue" to connect a proprietary system to our system. Both are real time in nature and transfer large amounts of data (a vision system.)

The vendor for the proprietary system has supplied a COM interface specification that we must develop our COM server to meet.

Everything in this system is Windows Servers or XP. The boxes are dedicated so we can dictate system policies and configuration.

The nature of the overall system is that the proprietary system and our own system may live on two different servers, so a remotable interface is necessary.

So I see three possible choices:

1) Develop a COM interface that speaks to our back end system via raw socket commands and responses.

2) Develop a COM interface that speaks to our system via Microsoft's RPC.

3) Implement the COM interface directly in our product. Rely on DCOM in order to provide remoting as needed.

My opinion is that the three choices above are in increasing order of system administration expertise and in decreasing order of programming complexity.

IE, #3 will require complex installation and administration and will require someone quite knowledgeable about DCOM security policies to deploy it. But implementing a COM interface directly in our product will involve no development of a back end interface and is simplest from a programming standpoint.

#1 and #2, much less complex deployment (specify an address, and that's it). But requiring development of a fairly complex PC to PC protocol.

Between #1 and #2, I would personally favor RPC because it replaces a ton of gruntwork in developing a socket based communication protocol.

We on this team have considerable collective experience with socket programming. I have some experience with RPC. None with DCOM.

Any other factors I should consider?
SW Designer Send private email
Monday, July 17, 2006
 
 
IMHO, avoid developing new COM code wherever possible.  We have to live with the huge legacy of COM that Microsoft has spawned but no more of this abomination, please!

You have to use COM to talk to your vendor's product (see above) but talk to your internal system in whatever protocol is most natural for its technology.
Mike S Send private email
Monday, July 17, 2006
 
 
What do you normally use to speak to your back-end system? If it's COM, use COM. If it's sockets, use sockets.

We made the decision several years ago that our back-end server will understand XML over HTTP and nothing more. I have written COM objects to call our server, and all they do is take the properties set by the user, shove them into an XML document, and HTTP them over to the server.

I can also call the server from a web page using a JavaScript/AJAX kind of thing. But it's still XML over HTTP. See the trend? ;-)
MBJ Send private email
Monday, July 17, 2006
 
 
#1 is the best for the reasons you stated (DCOM roll-out sucks). If you keep your protocol SIMPLE than it shouldn't be too much work.

I have to disagree with Mike S, that you shouldn't be making new COM projects. He calls it an abomination, but offers no reason why.

COM is great at what it does (letting components from different programming languages work together). What else is out there that lets my Excel macro use an object that was written in C++? CORBA?
Wayne B Send private email
Monday, July 17, 2006
 
 
Thanks for the responses.

Right now the back-end system is handled remotely within our current product stack using sockets. However, the protocol and much of the source code are undocumented. (the source is from an OEM partner.) Right  now we have no COM or DCOM interface within our product.

The COM requirement is coming from the external vendor who supplies the system we need to interoperate with.

Really high level protocols like XML-RPC, RPC over HTTP, etc are out because we have lots of image data to transfer. The protocol has to be binary level and suited to mass data transfer.

My thinking in favor of possible DCOM is that sticking a COM interface into the existing project would be the least amount of coding and the fewest moving parts. Yes, DCOM rollout would be painful, but we would avoid a lot of programming.

One thing I like about the RPC approach, though, is that "old school" RPC is tight and fast, and the interface can be derived from the same IDL (or a minor rewrite of it) that specifies the COM interface. I see raw sockets and RPC as similar in nature, but RPC is cleaner.
SW Designer Send private email
Monday, July 17, 2006
 
 
Have you checked out the RemObjects remoting library?  RemObjects remoting software is based on concept similar to DCOM (i.e., as I take it, extensive use of interfaces when communicating across processes and machines) but without using COM itself.  Uses binary transport (or others if you want) and is fast, without overhead or complexity of DCOM.

http://www.remobjects.com
Herbert Sitz Send private email
Monday, July 17, 2006
 
 
+1 MBJ. I'm no MS-basher, but COM is an abomination, particularly due to registry and system-wide versioning issues but also due to the black box component model (a component depending on another component) which bites you in the back when you can least afford it. Going with DCOM is an architectural buy-in on a poorly supported Microsoft whim; welcome to freeze timeout heaven, same with RPC, too much rigid configuration. XML over HTTP is very versatile, but I would try to avoid building your own sockets code from scratch to do it. I love the asynchronous feature of MSXML which takes advantage of Windows Internet proxy settings etc, but that's basically one-way.

That said, it sounds like you are a VB programmer, so disregard everything I said and go with what you know.
onanon
Monday, July 17, 2006
 
 
I would code the first version of the project in COM and then resources permitting, explore moving to XML over HTTP.

I'm specifically worried about the fact that if Microsoft, the vendor or any third party libraries change their specifications, you'll pretty much have to rewrite your code. One is bad enough, two can be a nightmare. If all three happen, you might as well run away to Aruba. Ideally, you should make it so that you're only dependent upon the vendor's specifications.

I suggested COM as the short term solution because a) you probably have a deadline and b) as bad as it is, there is a LOT of reference documentation on the web and you should be able to get up and running relatively quickly.
TheDavid
Monday, July 17, 2006
 
 
Use REST for the command and control channel then use raw sockets for transfering the image data over a separate connection. This is how ftp works.

If firewalls are problem then just stick with HTTP.
son of parnas
Monday, July 17, 2006
 
 
Err...  I misread the question. :(

Go with RPC as the short term solution - basically substitute RPC for COM in the answer I've written above. :)
TheDavid
Monday, July 17, 2006
 
 
We use MSXML on both clients and on the server, as well as XMLHTTP on the clients for posting the XML documents to the server. XMLHTTP honors Windows Internet proxy settings as mentioned and is easy to use from both VB and C++. It's been a win-win situation for us. Our server contains a little embedded HTTP server, but if we had to, we could port to IIS or Apache for added headroom. HTTP is the way to go for now.
MBJ Send private email
Monday, July 17, 2006
 
 
How do people secure these HTTP and low-level TCP/UDP solutions?  I'd never get them past security reviews and running in-house VPNs everywhere is nuts.
A. Hipster
Monday, July 17, 2006
 
 
I've worked on two very large real-time distributed  projects using DCOM and one not as large using sockets and found DCOM the easiest solution, if the machines are on a LAN. Then you can just turn authentication off for the specific components, and creating a component on a remote machine is as easy as doing it locally. The only recommendation though if you're sending lots of data is to forget all the COM marshalling and just stick your data in a big BSTR and send it that way (C-style casting it back to the structures or array of structures on the other end). That's not the official way but dealing with all of COM's custom marshallers and proxies and everything is too much of a headache.
Bill
Monday, July 17, 2006
 
 
Hipster, you put them on their own LAN and make it a closed system, especially if the servers are sitting in the same room.
Bill
Monday, July 17, 2006
 
 
"secure these HTTP and low-level TCP/UDP solutions?" I am no expert but don't you have the same or worse obstacles with RPC and DCOM? "VPN is nuts" really? it is a great start. Also HTTPS. For TCP/UDP you might be looking into PGP, your own PKI. But often you are talking about transfers within a LAN where security concerns are minimized behind the firewall.
onanon
Monday, July 17, 2006
 
 
My only issue with HTTP is that it adds performance overhead. The socket overhead is low, and RPC/DCOM are just a wrapper over socket. Do things with HTTP adds a heavy layer - IIS, and more. I have no idea about the scalability of this model. What will you do if you find that IIS can not handle large amount of data during the project?
Burner
Monday, July 17, 2006
 
 
HTTP is actually very light.

GET /some-file HTTP/1.0\r\n \r\n

To send data you use the header:

HTTP/1.1 200 OK Content-Length: 23983
son of parnas
Monday, July 17, 2006
 
 
Running a web server just to get data from one code module to another is ridiculous overengineering. If you need to remotely launch modules, use COM. If you have two dedicated systems, use sockets.
Len
Monday, July 17, 2006
 
 
Why not SOAP?  It succeeds XML over RPC and is backed my Microsoft or so I thought.

Is it that complex you need XML to describe things?  If it's much simpler, I'd go with simple UDP/TCP or as the 2nd poster stated, the more natural one for your environment.

Why is the vendor locking onto COM?  Already written?  (just curious)
~Eric
Monday, July 17, 2006
 
 
"The nature of the overall system is that the proprietary system and our own system may live on two different servers, so a remotable interface is necessary."

Do you mean remote as in two boxes sitting in the same room connected via an internal LAN or remote as in two boxes in separate states connected via the Internet?

There's a big difference.

If it's the latter, HTTPS or a VPN could be warranted. Or you could role your own.

If scalability is an issue, HTTP is the way to go as HTTP is all about scalability.

You don't have to use IIS or Apache. There are plenty of lightweight, embeddable HTTP servers you can use.
MBJ Send private email
Tuesday, July 18, 2006
 
 
Simple raw socket protocols are actually not terrifyingly hard to implement. They simplify a lot if you only have one connection to them, otherwise you can just start a thread per connection and optimise anything that needs it later.

If you stick to question/answer architectures and make everything synchronous, both ends' control logic gets trivial -- one end says "ask question, gather answer, loop" and the other end says "read question, work on question, write answer, loop"

If you use plain text interfaces, debugging it can use tools like "telnet" and you can record sessions and play bits back and so on fairly easily. Look at protocols like SMTP or POP3 for examples of how to structure operations with a protocol like this.

Once it's all working, it's easy to secure it with either SSL or a VPN.
Katie Lucas
Tuesday, July 18, 2006
 
 
I wouldn't use RPC. It's easy enough to get started, but the minute you need to use timeouts, asynchronicity, or you need to make minor changes to your protocol, it gets much more complicated.

Our product originally used RPC, and I could kill myself for making that choice. We suffered for a couple of years with poor performance over high-latency connections before I bit the bullet and rewrote everything with sockets. The end result gave a much better user experience, and is much easier to extend. I could have probably done some of the things with RPC, but it was much easier to get a good solution with sockets.
sloop
Tuesday, July 18, 2006
 
 
1. Sockets.
2. RPC
..
..
..
6.02 * 10 ^ 23.  COM/DCOM
Meganonymous Rex Send private email
Tuesday, July 18, 2006
 
 
Hi folks,

Thanks for the spirited debate. :) I never realized there were so many ways of looking at the issue. Well, wait, I guess I did...

I think we will use sockets for the following reasons:

1) RPC's issues.
2) We have no need to remotely launch the server. Both ends are dedicated. No need for DCOM, IOW. It's more like: both ends' applications will auto start, then the client will start searching for the server and will connect with it once found.
3) Any HTTP related transport is PURE OVERKILL and adds dependencies to the project. Plus, again, this is live video! We do not want to MIME encode large amounts of binary data.
4) Yes, this will be internal LAN traffic.

Thanks, all.

So my mind is made up. Don't try to change it. :)
SW Designer Send private email
Tuesday, July 18, 2006
 
 
I wouldn't dream of changing your mind!  For your particular application, live video, I'm surprised you'd consider anything else but..

"Any HTTP related transport is PURE OVERKILL and adds dependencies to the project."

Depending on the complexity of interchange, I would recommend HTTP -- or more specifically REST-style RPC.  You don't have to build in a web server -- HTTP is pathetically simple.  But it's a nice standard for handling a few basic tasks.

"We do not want to MIME encode large amounts of binary data."

Binary files are sent as binary over HTTP.  You just can't pack it into XML (so XML-RPC or SOAP is out) but plain pure boring HTTP is fine.
Almost H. Anonymous Send private email
Tuesday, July 18, 2006
 
 
Please don't use the term "raw sockets" when you aren't talking about raw sockets.
SomeBody Send private email
Tuesday, July 18, 2006
 
 
It'll be interesting to see how many of The Lame List gotchas end up enfeebling this application.  It is amazing how much bad TCP/UDP software is out there in production.

http://tangentsoft.net/wskfaq/articles/lame-list.html
A. Hipster
Tuesday, July 18, 2006
 
 
"Hipster, you put them on their own LAN and make it a closed system, especially if the servers are sitting in the same room."

Is this a serious suggestion?  Sounds like a case of "Trust the Force, Luke!"  Networks don't stay isolated, and any node is a potential router so you'd have to isolate the whole thing from the world.  Doesn't sound too useful.  Why not just skip the application and say you built it?  Or use one of those cardboard computers they display in furniture stores?


"'secure these HTTP and low-level TCP/UDP solutions?' I am no expert but don't you have the same or worse obstacles with RPC and DCOM? "VPN is nuts" really? it is a great start. Also HTTPS. For TCP/UDP you might be looking into PGP, your own PKI. But often you are talking about transfers within a LAN where security concerns are minimized behind the firewall."

Naive beyond belief.


Come on folks, you secure the sensitive resources, not the network.  Anything less is wishful thinking.  Home grown and cobbled up authentication and authorization schemes are just asking for it too.
A. Hipster
Tuesday, July 18, 2006
 
 
"Come on folks, you secure the sensitive resources, not the network."

So what does that boil down to, encrypting everything? 

"Home grown and cobbled up authentication and authorization schemes are just asking for it too."

I definately agree with this, though!
Almost H. Anonymous Send private email
Tuesday, July 18, 2006
 
 
The biggest limitation of DCOM is the way it handles network failures. DCOM has a hard coded timeout so it can be several minutes before a DCOM remote procedure call returns an exception. To work around DCOM limitations do NOT make a remote procedure call from your main thread. Also use a socket as an early warning of network problems. (Also design a generic DCOM interface that will not change and avoid chatty interfaces with multiple round trips to get one unit of work done)
Colin Sanson Send private email
Tuesday, July 18, 2006
 
 
"DCOM has a hard coded timeout so it can be several minutes before a DCOM remote procedure call returns an exception."

You can easily change it with DCOMCNFG. Across a LAN you never get timeouts like that, it'll return an 0x800706BE error if the connection is lost. The only delay is if you turn authentication for every call... so turn it off.
Bill
Tuesday, July 18, 2006
 
 
"It'll be interesting to see how many of The Lame List gotchas end up enfeebling this application.  It is amazing how much bad TCP/UDP software is out there in production."

Jesus!  How about: "If you plan on writing a solution that uses TCP/IP sockets, you might want to check out this list of common mistakes and/or misunderstandings regarding their use."
Meganonymous Rex Send private email
Wednesday, July 19, 2006
 
 
I think that last bit was meant as a caution against reinventing the wheel rather than encouragement to do so.
El Secundo
Wednesday, July 19, 2006
 
 
Yes.  Even informed good intentions don't guarantee a bug-free wheel.
Freling
Thursday, July 27, 2006
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz