A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.
I just recently finished working on a project that dealt with passing messages (through TCP/IP) back and forth between two systems. One of the major hurdles of the project was a shared header file that defined all of the message structures (in C++).
The overhead for this is that we spent a lot of time trying to debug problems when system A had a different version of the header file than system B and sent different sized messages. Also since the two systems had different operating systems we had to be concerned about big vs. little endian representations.
The plus side of this implementation is that we had a very well defined and compact message and was fairly efficient to send between the two systems once defined.
One thought that I had near the end of the project was to use an XML-based message structure to send messages. The pros of this is that the underlying structure of a message is just a giant string. This would eliminate a lot of the development problems we had with reading the wrong sized messages. Now we could just read until we got to some end delimiter and then send the message off. Also, we wouldn't have to worry about big vs. little endian representations any more, all of our numbers would be represented by characters (Which I guess is still a number at the end of the day).
The disadvantage to this is that now the message layer becomes much more complicated because to send a float across we have to create put it into a string and then the other system then has to decode it from string form back into a float.
I'm kind of torn between these two different approaches and am wondering from a design standpoint which may be better?
Thanks for any response!
>The disadvantage to this is that now the message layer >becomes much more complicated
No the message layer becomes much simpler - you call your std lib xml function to build the message and read it.
If you are worried about bandwidth gzip the messages.
If you are using XML anyway can you just use SOAP?
But put some thought into how complex the XML needs to be.
SOAP is not the best example of a clean design.
A readable text format is a great help when passing data around between machines, even if you end up compressing it into a binary stream - you can still more easily dump and debug it.
Take a look at todays codinghorror blog for alternatives to xml.
Monday, May 12, 2008
what is easier ?
create some method of checking the headers match at each end
creating helper classes to convert string to datatype
Monday, May 12, 2008
In a previous project we handled this by sending a protocol version in the initial connection handshake. So the server would get something from the client saying "I'm running version 123 of the protocol" and would send a message back to the client saying "I'm running 124." The version number would be kept in your message definition header, and depending on how you do things, you could just refuse to handle mismatches (we did that between internal nodes) or you could handle backward compatibility only (we did that between externally visible nodes).
The key is the convention that whenever you change any message structure, you update the version number in that same file. After a couple times, that becomes second nature.
The handshaking message could also contain endianness information, or I think we just defined a certain endianness and the client or server had to convert if they didn't use that natively.
Monday, May 12, 2008
XML is bloated no matter what you do. You could zip it but the downside there is the extra CPU overhead.
Still, at the end of the day I advocate ease-of-development over performance, unless you can prove your performance is real problem. That said, I've seen many XML-oriented solutions that were actually more difficult to work with than structs would be.
If you decide to use XML, make sure it's simple! :)
Tuesday, May 13, 2008
Simple XML or JSON. Float to string conversion is simpler than worrying about binary float compatibility between machines anyway since you have endian differences (note for international safety be sure to decide on a decimal separator rather than using a locale sensitive conversion). I don't know why someone mentioned bloat, since even XML will generally take less memory than a fixed field struct. Plus, by using something designed for datainterchange you'll have flexible field sizes, and the versioning freedom to add fields down the line without distrupting all agents. Avoid any form of "XML Validation" though.
Tuesday, May 13, 2008
I have solved a similar problem to this. I wrote a little code generator to do this, but the principle is the same. I calculate the protocol version by taking the MD5 hash of the file defining the messages. Sure this means changes to comments will "break" it to, but since this is automatically generated at compile time, its not something I would even notice, had I not written it myself.
Also, since I'm reading the "header", I generate the code to send the messages from the structure definitions, and endian-ness is taken care of by generated encoding/decoding functions(which, since you're in C/C++ would deal with an inadvertent compiler generated padding bytes in the struct too).
I would suggest you could use my solution, as I'm about to release it under an Apache license as open source, but its output is Java, so unless you wanted to write a C/C++ backend for me(wouldn't be too hard to do actually, since I already have an API and intermediate representation for the backends/frontends to communicate with, I won't be doing one anytime soon, since I just don't need it for my stuff).
Honestly, XML is half a dozen of one vs. six of the other. You still have to write a parse, albeit a more "abstract" one, to convert all those strings into the structs you use in the code, I'm presuming the structs exist because you *DO* something with them. XML is a great tool, it solves some "bikeshed" problems(Unicode, parsing API's, toolset to solve some "basic" problems) and does, in some sense, let you deal with "higher" level parsing problems... but its still parsing, unless you like using DOM objects instead of domain specific objects.
Thanks for all the responses.
I was actually inclined to originally post this on CondingHorror because of yesterday's post, however, I didn't think I'd get as much discussion from that site as I would here.
Based on the CodingHorror post, I tried to leave it open to an XML-based message because not all markup languages were created equal (XML was just the first thing that popped into my mind). The main point being that the underlying message is a giant string instead of a structure based on C/C++ data types.
Based on the posts, I am more inclined to try to implement the XML-based messaging due to the fact that we can have variable length fields much easier (Currently we're limited to 15 character names because that is how the message structure is defined). Development would also be easier since we're going to rely on the message layer instead of the communication layer to determine if a message is defined correctly (Hopefully this will lead to a more robust communication layer as well).
As far as bandwidth goes, in comparison XML-based messages will take up more relative bandwidth just because of the extra overhead of defining each field in the message (i.e. "name='12345'" instead of "12345"). Processing power is also not an issue since we're not working on an embedded system so hopefully we won't spend a long time trying to convert between floats -> strings and vice versa.
Thanks again for all the responses!
Good idea, I lean toward the: first make it simple to understand/test, then make it correct, then make it fast!
Plus you know that before this thing is even delivered there will be a requirement to add new fields and use it for different message structures that they haven't thought of yet!
One nice thing about all the redundant junk in XML is that it compresses very well and gzip on a modern CPU is faster than the network card.
The amount of information we're sending between the two systems isn't enough to worry about zipping. I think our largest message (using the C/C++ structures) is something around 250 bytes. Since we send a large number of messages as well, the compression would be a limiting factor since it would take much longer to compress and decompress a 32 byte message than it would be just to deal with it out-right.
Although if the messages did get exponentially bigger, compression may be a worthwhile activity.
I had to deal with this kind of thing fairly recently and I did it using a binary representation with variable-length fields that occur in a specific order. So to read something from C++, after receiving an entire packet (one of the transports is packet-based, so the thing works on a packet basis) you would say things like:
const char *str=reader.ReadString();
std::vector<const char *> strs(n);
...and so on, with some wrappers for higher-level types, and writing proceeding along similar lines.
This worked pretty well, I found, and whilst more hassle than the "send structs" approach it was certainly a lot safer, not noticeably slower (if at all) and made it easy to sort out the result using pretty much any language. Each type of packet has an ID, so provided people remember to update this when appropriate then any mismatches are easily spotted, or handled invisibly.
Compared to XML, or indeed any text-based format, it is a much smaller amount of code, a much smaller amount of data, and (with a modicum of thought, a fixed maximum incoming packet size, and use of blocking sockets for outgoing data) can be done without needing to dynamically allocate any memory. These constraints are not relevant to everybody, but they are to me, and in any event I've found sticking to them beneficial.
I personally am quite a fan of the binary representation, because it is quicker to encode/decode, requires no external libraries that may allocate memory behind your back and contains no unnecessary information. You can implement the above system in about an afternoon, so it's not like you're paying a big price to get these advantages either!
The result is a bit of a pain to examine by eye, I do admit, and of course the lack of metadata means that you have to keep your wits about you when changing the packet format, but I found neither of these a problem in practice, and in any case suspect that useful metadata could be added without imposing a significant penalty.
If you and your coworkers have been through the making-cross-platform-structs-work "thing" a bit, you should find this sort of approach no problem, providing a large fraction of the same benefits with fewer day-to-day problems. Of course, I'm in my shoes, not yours, so, pinch of salt and all that...
Tuesday, May 13, 2008
Whatever happened to Sun RPC?
All this talk about encoding data structures over the network and yet no one has used the words 'marshal' or 'unmarshal' yet?
Java has a built-in serialization that can accomplish this without any extra code for each data structure. But it's completely tied in with Java and it's got annoying limitations.
XML is bloated, inefficient and ugly. But for better or worse it's an accepted standard.
One of these days I'd like to build an encoding scheme that encodes the data structure within the serialized data stream. It would have to runtime-bind all data types. There'd be the usual scalar primitives plus string, vector and dictionary, all derived classes of a root class, and each object can queried at run time for its specific type.
Tuesday, May 13, 2008
You can do xml-rpc, the best of both worlds ;-)
I remember these sort of things being the latest fashion for instrument/experiment control, we went through generations of them which would let our VAXes talk to our Norsk datas and our Sparcs. They had all the hooks to handle machines with different number of bits/byte etc.
In the end one letter commands, simple text messages for small values and dumping binary data files to a shared disk was about the only thing that ever got implemented.
This topic is archived. No further replies will be accepted.Other recent topics
Powered by FogBugz