The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Impact of open sockets on scalability?

Hi,

I'm designing a client-server communication protocol and I'm wondering what kind of an impact the number of open socket connections have on a server. Specifically, can one open millions of socket connections from different clients to the same server if each such connection does little/nothing? Does the number of open connections limit scalability in a big way? Is there a practical limit on the number of open connections imposed by the operating system, TCP/IP, firewalls, etc?

This connection is meant to allow the server to send asynchronous messages to the client (for example, notifying it that the server is rebooting) as well as allowing it to track when clients log off (by disconnecting the socket).

Please advise.

Thank you,
Gili
Gili Send private email
Wednesday, July 09, 2008
 
 
"millions of sockets" does not sound good.

http://www.google.com/search?hl=en&q=number+of+sockets
da
Wednesday, July 09, 2008
 
 
I wish I could remember it, but I know *nix has a limit, but it can be configured.  Windows also has a limit, and I believe it is hard-coded (and lower).

Each socket will naturally need some sort of data structure to represent it.  Millions is probably out of reach.
Doug
Wednesday, July 09, 2008
 
 
dev1
Wednesday, July 09, 2008
 
 
The port number can only be guaranteed to be a 16-bit number, maximum value in the tens of THOUSANDS. Since each socket needs a unique port number, MILLIONS of sockets is simply not an option. While I doubt that the memory required for the socket entries would be a significant limitation, the need for adequate CPU resources to service all those sockets, which would be a challenge (at least), even on a modern, multi-gigahertz, multi-gigabyte, multi-core system.
Jeffrey Dutky Send private email
Wednesday, July 09, 2008
 
 
You can maintain a million open sockets if you design your application to be distributed. You just can't have a million sockets open on one single server.
Wayne Bloss Send private email
Wednesday, July 09, 2008
 
 
>> Since each socket needs a unique port number, MILLIONS of sockets is simply not an option <<

Technically each connected socket needs a unique combination of client IP, client socket, server IP, server socket, so the limit is not the size of the socket field if each connection is to a different IP address.

However, I agree that an architecture requiring a server to maintain millions of sockets simultaneously is likely to not work well (or at all).
mwb
Thursday, July 10, 2008
 
 
While platform dependent, each open socket is likely to eat some non-paged memory.

The Windows limit is not fixed, and I seriously doubt it is "lower than Unix."  TCPNumConnections defaults to the max (0xFFFFFE) in Win 2003 Server.
Codger
Thursday, July 10, 2008
 
 
I guess I should have made the question a bit more realistic... Realistically speaking, how many clients does a server handle simultaneously (due to other limitations such as bandwidth, cpu or memory)? My guess would be thousands.

So my question is: will maintaining 2x as many open sockets on a server impacts its performance in a noticeable way? What is the cost of an open socket? Is it mostly a matter of bandwidth (for keep-alive)?
Gili Send private email
Thursday, July 10, 2008
 
 
I would expect the cost to be highly dependant on the application. Having two times as many idle connections probably won't be a big impact except reduce resources for active operations. Having twice as many active connections doing expensive work will, of course, make an impact. So I would guess that if you're asking this question, you're invetigating the wrong limitting factor for your application.
Benjamin Manes Send private email
Thursday, July 10, 2008
 
 
An open socket with no traffic on it is pretty close to zero CPU overhead for an application. It's also pretty close to zero for the kernel.

Why?

Because these things are event driven. Activity on network interface causes activity on a device driver which causes the kernel to service the request and flag the socket as readable or writeable -- and if you're using something like epoll, you just get access to the same flags.

{If you're using select, you've got setup/teardown cost on the fd_sets -- but anyway fd_sets usually aren't big enough to do large socket arrays with. Select doesn't scale well. Use epoll.}

You'll need some RAM for the connection -- but not a huge amount for a socket with no activity. Nothing that you really care about.


How many can you sensibly have? I've seen applications running 50,000 bittorrent connections pulling the data off disk which can saturate multi-gigabit networks -- all running on regular 64bit server hardware.

Don't underestimate this -- the guys who wrote it are smart; and they've spent several years finding and removing bottlenecks.

But it is massive performance from relatively cheap hardware. You really do get some impression of how powerful modern computers are and it also tells you how utterly shite most modern applications are that they can take that sort of power and suck it all up leaving none for the user interface.

So the answer to your question is that tens of thousands of sockets are definitely doable. Your application, if properly written, will probably be limited by one of two things; either its ability to pull data off the disk OR the capacity of your network interfaces.
Katie Lucas
Thursday, July 10, 2008
 
 
Assuming Windows as that's what I know. An open socket consumes a small amount of 'non-paged' pool and this is a finite resource that scales with the amount of memory that you have in the machine. Each pending read or write may also consume some 'non-paged' pool and/or use up some of the 'locked pages limit', and so the more activity you have on a socket the more resources it consumes. The standard way to reduce the amount of 'locked pages' that are used by a 'lots of connections, not much data flow' application is to post 'zero byte reads' and then issue a real read when the zero byte read completes, this allows you to only use resources (both normal memory and non-paged pool) when there is data to read rather than all the time.

I've done quite a lot of work in this area and I have a free server framework that might get you started. It uses IO Completion Ports for overlapped I/O on 'lots' of connections with very few threads, performs well and does all of the difficult stuff for you.

Though I haven't tried 'millions' of connections, I have run some tests that achieve more than 70,000 active connections on a Windows Server 2003 machine with 760MB ram. See here: http://www.lenholgate.com/archives/000568.html for more details on that and some test tools that might give you some ideas.

I've also stress tested servers and pushed them to exhaust these finite resources ( http://www.lenholgate.com/archives/000570.html ) and what I've now started to do is focus on being able enforce limits on resource usage either through limiting the number of connections to a number that you've tested your server configuration with ( http://www.lenholgate.com/archives/000580.html ) or by making sure that you control the amount of data you're sending and buffer it yourself rather than relying on the TCP stack to do it for you ( http://www.lenholgate.com/archives/000788.html ) or both. The reason that this is important is that the 'non-paged' pool and the 'locked pages limit' are machine-wide, finite resources. Using them with abandon is bad for your box. Often it will be your application that starts failing when these limits are reached but I've had reports from clients that suggest that poorly tested NIC drivers can bluescreen a machine if you exhaust 'non-paged' pool.

Anyway, having said all that, I expect you're using a Unix platform ;)

Just remember that getting to the point where you CAN write something that is massively scalable and performs well is only the start of your problems ;)

Len
Len Holgate Send private email
Thursday, July 10, 2008
 
 
"Just remember that getting to the point where you CAN write something that is massively scalable and performs well is only the start of your problems"

That's an insightful (and highly quotable) comment, indeed.
Pablo Send private email
Thursday, July 10, 2008
 
 
As a semi-tangential note, you might want to consider that a communications protocol that supports the requirements you mentioned (asynchronous messages being sent to the client, tracking when clients log off) already exists, so there may be need to design a new one.  Unless there are further requirements you didn't mention, have you considered using the eXtensible Messaging and Presence Protocol ( http://www.xmpp.org )?.  There are multiple conformant open source client and server libraries out there that you might be able to repurpose to your needs.
Greg
Thursday, July 10, 2008
 
 
Sorry, that last message should read "may be *no* need" rather than "may be need".
Greg
Thursday, July 10, 2008
 
 
If you're using Java, JMS implementations (e.g., ActiveMQ) might be a good choice.

The only good thing about sockets (and it can be a VERY good thing in some cases) is that you get OS support for dealing with situations where the connection is broken.  These are not bulletproof though, so most HA apps also implement some kind of "heartbeat" even with sockets.
BillAtHRST Send private email
Thursday, July 10, 2008
 
 
PS: http://support.microsoft.com/kb/929851 indicates that the default number of system-wide open sockets for Windows Vista is 16384 and my tests seem to confirm that. You can play around with this number but the hard limit per server (regardless of the OS) is 64k and probably more likely < 60k

Makes one wonder if TCPIP will allow more than 64k ports one day, now that P2P is mainstream.
Gili Send private email
Thursday, July 10, 2008
 
 
My mistake, the above document talks about the number of *outgoing* connections, it says nothing about the number of incoming connections a server may handle.
Gili Send private email
Thursday, July 10, 2008
 
 
If you use UDP sockets can't you multiplex all
your traffic off the same single socket?
Object Hater
Friday, July 11, 2008
 
 
> can't you multiplex all your traffic off the same single socket?

Yes. Regarding 16-bit port numbers (ignoring any OS-level limitations) I'd think you might get > 64K ports on one machine by homing more than one IP address.
Christopher Wells Send private email
Friday, July 11, 2008
 
 
>> You'll need some RAM for the connection -- but not a huge amount for a socket with no activity. Nothing that you really care about. <<

IIRC, each open socket consumes about 2k of RAM on Windows.  So a million open connections is 2gb of RAM, just to track socket handles, and doesn't include RAM used as buffers for incoming/outgoing data.  That's a non-trivial amount of memory being used.

The other thing to think about is that's your steady-state condition.  If you have a network hiccup and drop connectivity, all the clients are going to try and reconnect at about the same time, so you'll have a huge peak load condition.

One more thing to think about -- what happens when you need to take the server down for maintenance?  All the connected clients either go down, or need to have a way to transfer to a backup server.

My advice is to switch to something stateless (like HTTP), so that you can scale your app horizontally with load-balancing hardware and software.  This spreads the daily load around, any peak load also being spread out, and you can do rolling maintenance outages on your servers with little impact to clients.

Since you want to monitor when clients go offline, a heartbeart would be the way to approach this -- have each client send a short "I'm alive" message twice a minute.  So you'll know when a PC goes offline with a granularity of 30 seconds or so.
xampl
Friday, July 11, 2008
 
 
It’s not feasible to have a system where a server based application runs on 1 box with 1 NIC and 1 IP address and will accept millions of simultaneous connections... 

As others have said, you’re first off facing the 16 bit limitation of port numbers per tcpip connection.  I would say that the amount hosted by an IRC server is the best that can be expected. 

You really need to rethink the entire process and answer these types of questions.

-Is the information realtime?  How long a delay can be tolerated?
-Is the connection State-full or Stateless
-Do you need lossless data delivery?
-Is there a firewall on ether side?
-Who initiates the connection? 
-What happens if the connection is interrupted?

Depending on the answers, you might not really need to maintain a million open sockets, but rather have a million rows in a database where you remember what happened the last time user number  999,999 connected to you.

Without answering these questions, you could be vastly over engineering or under build the system..

The decision tree might go like this:

If the information is realtime then TCP socket is not a good idea.  Why? If there is an error and a retransmit occurs, the recovered packet will not be needed by the other end.  But the TCP socket will keep trying until the data is delivered..  Think about Skype. If the connection goes down, do I really want to hear what was said, 3 seconds ago. 

Next, does the client ALWAYS ask for information and the server ALWAYS sends it back?  Or does the server ever need to send data Async to the client?  If there is no Async communications requirements, then a resource constrained statefull protocol is not needed.  The client can connect to the server as frequently as it wants and it just needs to present its ID number.  This type of system can be either TCP or UDP..

But if the server does send back information async and the client is behind a firewall, you must have a TCP socket synchronous connection..  Because if an error occurs, there will be no way for the server to get to the open port with the client behind the firewall.

Etc,etc,etc

So, my suggestion is to first analyzes the problem and answer the above questions.. If you can make the system work as a webpage with a userid in the calling URL, do that first.  You just shaved months of design and testing off the schedule. Next if you really want to maintain 1000's of open connections, download and start hacking the source code for jabber or IRC...  They must has spent 100's of man years optimizing those systems.. 

Another major problem is: How can you ever test a system that has millions of connections??  You cant easily. 

So, try to think about building a system that has 1000's of connections and then go horizontal (buy more boxes) when the traffic comes..

Good luck..
mb Send private email
Friday, July 11, 2008
 
 
You also might want to review this

http://www.laynetworks.com/Comparative%20analysis_TCP%20Vs%20UDP.htm

They have a discussion on the benefits of TCP vs UDP
mbalsam Send private email
Friday, July 11, 2008
 
 
Gili -

The number of port numbers does not limit the number of open sockets, only the number of open sockets between two individual systems.

If I have systems A, B, and C, A and B can *both* have up to 64k sockets to C.  C can reuse local port numbers, as long as the remote source port and IP are different.

So if A has IP address 10.0.0.1, B has 10.0.0.2, and C has 10.0.0.3,
then you can have (with notation IP:port) sockets 10.0.0.1:10000<->10.0.0.3:10000, 10.0.0.1:10001<->10.0.0.3:10000, 10.0.0.2:10000<->10.0.0.3:10000, and 10.0.0.2:10001<->10.0.0.3:10000 all communicating at the same time.

You just need to remember to use SO_REUSEADDR with the socket so you can get this.  Try it!
Joe Malicki
Saturday, July 12, 2008
 
 
One million sockets on one server is infeasible, but 10,000 is possible with careful design and a lot of tuning. This page has a ton of information on how to design a web server to handle many clients. The API notes are targeted toward *Nix, but the designs should apply to any platform. Even if you decide to use somebody else's webserver rather than writing your own, it's probably helpful to understand the reasoning behind how to make it scale.

http://www.kegel.com/c10k.html
CS Student
Sunday, July 13, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz