The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Threaded I/O on Linux

I come from a Windows background, and have recently begun writing a TCP/IP based server program for Linux.  In trying to achieve the best performance in my server (which will likely need to support a large number of connections concurrently) I am researching various ways to achieve something similar to I/O Completion Ports on Windows.

My experimentation with various techniques leads me to believe that I may be better off avoiding threads in favor of the classic fork/prefork type server.  One thing I'm running across with epoll, for instance, is that the system wakes all threads blocking on epoll_wait, which does not seem desirable.

Can anyone point me to a resource that might discuss the design of high performance network servers on Linux?
Is it worth it? Send private email
Monday, October 20, 2008
Start at the beginning

You can use select() to service many clients without the need to spawn a thread/process for each.  This *may* be a good choice, depending on your application, # of concurrent connections, etc.
Monday, October 20, 2008
First, I'd just google on "linux network server threads programming" and see what comes up. I expect that there are lots of good on-line discussions and references.

My understanding is that multi-threading in network servers is currently out of fashion (on linux, at least) for some reason. Every time I have seen this topic raised, examples are cited showing that single threaded servers perform very well in many applications and that multi-process servers (such as Apache) are usually better than multi-threaded servers for the same application. I haven't paid enough attention to say what the technical justifications are.
Jeffrey Dutky Send private email
Monday, October 20, 2008

My Googling didn't net me a whole lot of useful info. 

I suspect that you are correct, though.  My investigation into this issue leads me to believe that, at least on Linux, there is no benefit to using threads versus multiple processes and in fact there are some drawbacks, such as the one I mentioned.

I also found some information related to the epoll_xxx APIs that leads me to believe that there isn't a good way to use threads with them.  Finally, the aio_xxx family of functions (Again, Googling) do not seem to perform asynchronously with sockets, negating any potential benefits.

So: I think that the best approach on Linux is to use the Apache prefork method, with each process using something like poll() to multiplex connections.  In this fashion it would probably be easy to write a server to handle lots of concurrent connections across 'n' processes.

However, I wanted to double check on JoS 'cause there are so many smart people around!
Is it worth it? Send private email
Monday, October 20, 2008
The only advantage I could see to multithreading is that it may simplify IPC a little bit (since all threads share the same memory space). If you have to push lots of data to the worker threads (whether they be threads or separate processes) then you will either need to use shared memory (mmap) or you will need to use actual threads. In either case you'll need to handle the coordination of simultaneous access to shared memory, which may end up being your real bottleneck.

There is a section in the current edition of Stevens' "Advanced Programming in the UNIX Environment" that looks at performance of various IPC methods. I think it compares Mac OS X, Solaris and Linux (but I don't have the book to hand right now) and may be of help to you.

You're right about googling for this: I'm not turning up any good on-line references with simple search terms.

Of course, the old adage about optimization applies here: don't do it until you know you need it, then experiment. You could easily write a couple of implementations to test threads vs. processes vs. single server and see which does better. I'd try writing a single server implementation and only go to the more complex versions if/when the single server proved to be too slow.
Jeffrey Dutky Send private email
Monday, October 20, 2008
Why don't you do a microbenchmark where you create 1000 threads, and create 1000 forked processes?  I have a feeling the threaded method scales much better.
SmartYoungAnonPoster Send private email
Monday, October 20, 2008
You can't judge the performance of real code from microbenchmarks.
Monday, October 20, 2008
A thread pool with epoll is the way to go.
Monday, October 20, 2008

1000 of anything is probably going to perform like crap.  Remember that on a machine with N processors, only N "threads" will ever *actually* be running at a time.

The optimal number of threads/processes to run probably depends very much on the type of work being done in the server.

The real issue at hand is whether Linux even allows the programmer to write a multithreaded program in a good way (when it comes to socket I/O.)

Currently, Linux does not seem to support true asynchronous I/O for sockets, and the efficient polling mechanism provided by epoll() does not play well with threads: multiple threads blocking on an epoll_wait() all wake up, resulting in the so-called "thundering herd" problem.
Is it worth it? Send private email
Monday, October 20, 2008
You can use a single thread to wait and that thread dispatches work to the thread pool.
Monday, October 20, 2008
select() has a fairly low limit on the number of descriptors it can monitor, and like poll(), scales poorly for large numbers of connections.  The key problem is that all the descriptors you're monitoring need to be passed from user to kernel each time you call select().  [The same goes for poll().]  Then, to check for descriptor "readiness", you need to loop the whole set which means it's O(N) efficient.

With epoll(), the descriptor set is maintained in the kernel and when descriptors are ready for work, the notification is O(1) efficient. 

For a single-process server, using epoll() is better across the board than select or poll and is probably *easier* than using select().  It is not portable to other UNIXes though.

One approach you might take is a hybrid approach in which a process forks some number of times and each child multiplexes using epoll().

This would allow for true concurrency on a host with multiple processors, would scale well to hundreds of connections, with the drawback being that shared memory and some mutual exclusion would be necessary in the case where the processes would need to share data.

See Davide Libenzi's original page on EPOLL, it's a good source of info:

Dan Kegel's page has some info on this topic too:
Tom Dial Send private email
Monday, October 20, 2008
Use epoll. Put FDs into epoll groups. Only wait one thread on each epoll group to catch events.

When an FD wakes that thread, it can wake a worker, hand the FD to it, reset the flags it's waiting for (so it doesn't get woken again) and go back to the poll to see what else is happening.

The worker does what it needs to do on the connection. When it's finished, it changes the epoll wait flags, and goes back to sleep.

The rule is that only catcher threads remove flags, only worker threads set them.

This scales to handling thousands of connections with no problem.
Katie Lucas
Tuesday, October 21, 2008
The best way to handle multiple clients I/O is the way many HTTP servers do it:

1) have a single thread which accepts the incoming data.

2) dispatch the job created by the incoming data to one of the available threads from the thread pool.
Achilleas Margaritis Send private email
Wednesday, October 22, 2008
"have a single thread which accepts the incoming data."

There's a bottleneck right there.

Accepting the data can take a decidely non-zero amount of time (particularly if you're accepting data rather than just the connection. You'd be amazed how often client systems open the connection, say "hi... erm... hang on a mo... er.. yeah.. I wanted something... erm....")

There's lots of ways that thread can get saturated.

Racing accepts work fine on Linux; exactly one accepting process/thread will be woken for an incoming connection. That solves this saturation bottleneck.
Katie Lucas
Thursday, October 23, 2008
>> There's a bottleneck right there.

Are you sure? If the thread dispatches the data right away, by passing a pointer with lock in place, I can't imagine how this thread can't keep up with the network driver.
Thursday, October 23, 2008
"If the thread dispatches the data right away"

  int myfd = accept(listensocket);
  MyMessage *mymessage = new MyMessage;
  CThread *threadp = findAFreeThread();

If your loop looks like this, you're presuming that the message read will complete before more connections arrive. Network interference arrives in batches; while your network only loses (say) 1% of packets, it turns out that that means it loses 100% of packets 1% of the time. Those retransmits will cause stalls in your loop -- and will hence stall accepts.

Typically, unless you've set it otherwise, you only have an listen queue that's a few sockets long; more than that stacked up will get refused.

It will also fail under the following scenario -- a typical way to smoke test a webserver is to call it up with telnet and type "GET / HTTP/1.1" at it.

In the case of this loop, it will stop accepting new connections while you're looking for the / key, because it's blocked in the read.
Katie Lucas
Friday, October 24, 2008
We have a server written in Java that handles hundreds of connections at a time. Our approach is to use a single "listening" thread. When a connection occurs it hands the connection to a worker thread from a thread pool and then goes back to listening for the next socket connection. The listening thread does NOT read the data from the socket. So the listening thread does very little and can get back to listening as soon as possible. It is the worker thread that receives the data over the socket, processes it, returns a reply, and closes the socket.
Friday, October 24, 2008
The OP obviously is concerned about the asynchronous mode, which is what high performance servers are all about. All my comments here apply to asynchronous IO only.
Friday, October 24, 2008
>> The OP obviously is concerned about the asynchronous
>> mode

Sort of, although strictly speaking,  I don't think Linux *does* asynchronous network I/O.  I have read that the aio_xxx functions do not actually operate asynchronously on Linux with sockets, and as I understand it, epoll is basically the same as poll except that scales better due to the fact that the descriptor sets reside in kernel space AND performance is O(1) because the kernel doesn't scan the descriptor table to find ready descriptors.  (May be off on this)

What I'm really interested in, is what are the best practices on Linux, and I think I got some great tips here from Katie Lucas et al. 

I plan to perform a few experiments with competing models (perhaps using a simple ECHO server or something) and if anyone's interested I'll post back with what I find.

As far as asynchronicity, it may or may not matter whether Linux has truly asynchronous I/O, it may still perform as well or better as systems that do when the programs are written appropriately.
Is It Worth It? Send private email
Monday, October 27, 2008

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz