The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Designing a UDP streaming mechanism - getting through firewalls

I am designing an application that needs to use UDP to transfer data from one point to another across the internet. (It is a streaming media application.)

The architecture that we intend is basically: windows client applications speaking in full duplex to each other (using UDP)  facilitated by a reflector application in the middle (probably running on a Linux server.)

The big question I have is concerning how a client that is inside a firewall can receive a series of unsolicited UDP packets from outside the firewall.

The logic of sending UDP packets from a sending windows client to the Linux reflector is pretty obvious. What I don't get is to how to forward those UDP packets from the reflector to the other Windows client and how to get through the receiving client's firewall.

I understand socket programming over TCP well enough. I know that opening a socket based connection creates a 'hidden' socket server on the client with a concocted port number, that listens for responses to its own requests sent to the 'real' server. I also know that firewalls understand this mechanism and automatically poke holes through the firewall to permit responses to come through.

I think what I am looking for is understanding of how the listener for UDP packets on the inside of a firewall can receive those packets in a similar manner.

Thanks for any advice.
Bored Bystander Send private email
Sunday, July 22, 2007
 
 
PS: one of the goals of this design is that the user does not have to manually configure their firewall. I am looking for a way to allow unsolicited UDP traffic to get into a firewall from outside.
Bored Bystander Send private email
Sunday, July 22, 2007
 
 
Have you looked at multicast? It uses a specific IP address for publish/subscribe type operation.

The firewalls would all have to be capable of the control protocol (IGMP?)

Here's a starting point;
http://www.hep.ucl.ac.uk/~ytl/multi-cast/index.html
sgf
Sunday, July 22, 2007
 
 
I highly doubt that you will be able to "fool" firewalls in this way. Firewalls are specifically designed to prevent such traffic (this is one reason for the failure of DCOM, and its replacement with the much less efficient web services). In particular firewalls are designed to prevent unsolicited incoming traffic.
Dan Shappir Send private email
Sunday, July 22, 2007
 
 
You can't really do this. The entire point of a firewall is that it blocks out all traffic not initiated from the inside. :)

The best you can hope for is that the firewall is stateful[1], in which case bidirectional UDP traffic is okay, as long as it's initiated by the client behind the firewall. Most firewalls these days seem to be stateful.

[1] See http://en.wikipedia.org/wiki/Stateful_firewall
R
Sunday, July 22, 2007
 
 
If you want to go through a firewall, the client is going to have to initiate a TCP connection with your server.  It is technically possible to get UDP to forward through a firewall but your users would need to actively work with whomever manages their firewall to get it to happen and it probably won't scale well to multiple users behind a corporate firewall.

Using TCP allows your users to initiate the connection, but even then that can generally only do so over port 80 because other ports are frequently blocked in a corporate setting.  Using other ports will (generally) require corporate assistance from whomever controls the firewall.

Even if you use port 80, you are still likely to get banned by the admins if your app is at all successful.  That's because many of those stateful firewalls that other responders have mentioned are optimised for the type of short lived connections that web browsers typically make.  If you start to have 100 users begind the firewall holding ports open for long periods of time, you'll wedge the firewall and prevent other users from surfing the web.

And yes, big companies do have special case hardware and handling in place for the major IM clients (which generally have TOS requirements that prevent you from piggybacking on them).

The C++ code for what you are talking about is cake.  It's the deployment that is the hard part.  And don't even get me started on .NET remoting and WCF - I'm generally a major Microsoft fanboy but they completely screwed the pooch with those architectures because neithe one has any ability to exist in a typical distributed environment with machines living behind NAT routers.
system of a Don Send private email
Sunday, July 22, 2007
 
 
Thanks for the comments so far. Just to clarify:

I do not expect to be able to just fire UDP data at a firewall protected PC and get through. I understand the concept of firewall statefulness.

I am not concerned about corporate practices or corporate firewalls. This just has to work with SOHO equipment and "loose" SOHO type practices.

What I am really trying to do is set up some kind of an IP session between a client behind a firewall, and a server outside the firewall, and then be able to freely send UDP data to the client. UDP because I would rather lose data than have a protocol like TCP/IP waiting for data that never arrives.

And I would much rather have the server outside the firewall be the initiator and send the data as it is available, rather than have the client "poll" (and thereby 'pull') the data by requesting it at intervals.

The only code examples I see about UDP are associated with sending data with no concept of a connection. And again, obviously, this won't fly.

How do streaming radio stations do this? Obviously they get through consumer firewalls.
Bored Bystander Send private email
Sunday, July 22, 2007
 
 
Update: I found out about "RTSP" - Real Time Streaming Protocol - the foundation of most streaming stuff like RealAudio, WMP, etc. Now poring over ftp://ftp.isi.edu/in-notes/rfc2326.txt.
Bored Bystander Send private email
Sunday, July 22, 2007
 
 
Adam
Monday, July 23, 2007
 
 
I used Wikipedia to freshen my memory: "UDP does not guarantee reliability or ordering in the way that TCP does. Datagrams may arrive out of order, appear duplicated, or go missing without notice."

Do you have to do a lot of work to live with that messiness?
Stan James Send private email
Monday, July 23, 2007
 
 
I don't remember the name of it, and it might only work for TCP, but I remember reading an article about how Skype works in it's P2P mode.  Basically both clients try to open connections to each other (even though the other side's firewall will block).  However, the local firewall sees the attempted outgoing connection (which ultimately failed at the remote end) and then allows data in from that remote port (because it assumes the outgoing connection succeeded).
Doug
Monday, July 23, 2007
 
 
>> How do streaming radio stations do this? Obviously they get through consumer firewalls.

They use HTTP.  It's really the only semi-reliable way of getting through firewalls.  It's a common fix for streaming issues in Windows Media Player to disable RTSP in the Network settings forcing it to use HTTP.
SomeBody Send private email
Monday, July 23, 2007
 
 
In general, unsolicited packets, whether UDP or TCP, do not get past a firewall. That is the whole point of a firewall - unsolicited packets don't get through.

However, there is a way. For any conversation with a remote host, it must be possible for it to reply when you send it a packet. So if you send a packet to any computer, that makes it possible for it to send a packet to you - at least for a little while.

If two computers behind a firewall are in contact with a server, they can be given each other's addresses. If they then both attempt to send packets to each other, the firewalls will allow the packets through. This even works through NAT.

Duncan

Monday, July 23, 2007
 
 
See here for some example NAT traversal code that may help (I haven't looked at it, but I at least consider its author worth listening to), along with a good explanation that renders irrelevant any summary I might provide here:

http://www.speakeasy.org/~hplus/libintroduce.html
Tom_
Monday, July 23, 2007
 
 
Do a Google search on the STUN protocol. This deals with two aspects of getting UDP traffic through a firewall:

1. The outgoing STUN traffic will allow reciprocal UDP traffic for a short while on most stateful firewalls (just keep repeating the STUN query as needed).

2. The response from an external STUN server (outside the firewall) allows your application to determine if its internal IP address/UDP port is being mapped by NAT (most SOHO routers combine firewall and NAT functions). Of course, a better alternative is to design a NAT friendly protocol in the first place: no IP addresses or port number in protocol content - receiver replies to IP address port in the received IP header.
Stephen C. Steel Send private email
Tuesday, July 24, 2007
 
 
> I think what I am looking for is understanding of how the
> listener for UDP packets on the inside of a firewall can receive
> those packets in a similar manner.
> The only code examples I see about UDP are associated with
> sending data with no concept of a connection.

Since everyone else has helped you solve the firewall issue, I'll just answer your UDP questions.

Communication with UDP datagrams is inherently connectionless with regards to UDP's implementation as a transport layer.  That doesn't mean that connections cannot be implemented by application code using UDP datagrams/sockets.  For example, Trivial File Transfer Protocol (TFTP) uses UDP sockets & an application state machine to form connections beween server & client.

So you just develop your server-side UDP app to send UDP packets to clients & your client-side UDP app to wait for packets to process.


Am I missing something?
Ian Johns, author of µC/TCP-IP
Wednesday, July 25, 2007
 
 
Not just firewalls. Routers can sometimes cause failures of connectionless protocols, such as UDP. I'm not an expert so read up on this or find a real expert.

Start by reading about Network Address Translation.
frustrated
Wednesday, July 25, 2007
 
 
If you have a "reflecter" server, you should be fine with stateful firewalls.  You don't need to use TCP.

Break it down into two clients connecting to a server.  The server just happens to tie together the clients.

Client A connects to the server and the server sends packets back.

Client B connects to the server and the server sends packets back.

The server just happeneds to forward packets between client a and client b.

When testing, just pretend that client B is abstracted away into the server.  After you have the connection to the server working, reuse your server code for client B and tied the two ends together.

Wednesday, August 01, 2007
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz