The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Why is TCP so mucher slower than native file transfer

I'm working on code to transfer files using TCP and want to get an understanding of why it is quite a bit slower than using Windows to copy the same files. I'm using two PC's on a WLAN. Any pointers to articles appreciated.

Related to this are there any ways to speed up TCP?
Neville Franks Send private email
Tuesday, August 15, 2006
 
 
I'm assuming Windows ...

Without knowing what network protocol implements file transfer (Google for "samba" or "smb"), one reason may be that when you write TCP you're probably writing user-mode (not kernel-mode) software.

It's also possible hat you haven't implemented it well (e.g. you're using small buffers, or doing something else which uses a lot of CPU).

> Related to this are there any ways to speed up TCP?

Try WSASend (or TransmitFile) and WSARecv, and use large buffers. I haven't tried it but I remember long ago someone saying they were able to nearly saturate their wire (i.e. occupy most of their LAN's bandwidth).
Christopher Wells Send private email
Tuesday, August 15, 2006
 
 
Thanks Chris. I've using Windows. My app is usr mode and using WinSock2 with 64K buffers. I don't think the CPU is doing anything much. The socket code is using send() and recv() not the WSAxx functions.
Neville Franks Send private email
Tuesday, August 15, 2006
 
 
Are you sure it's not just that your send call doesn't return untill all the data has been sent while the smb copy returns as soon as the file info has been sent and the actual data copy continues in the background?

This was a problem with NFS2/3 by default one of them waited fro the file to be physically committed to disk at the other end while the other returned immediately.
Martin Send private email
Tuesday, August 15, 2006
 
 
>Are you sure it's not just that your send call doesn't return untill all the data has been sent while the smb copy returns as soon as the file info has been sent and the actual data copy continues in the background?

I'm comparing it with a backup type app that copies files. It doesn't end until the files are copied.

I need to read up on SMB I guess. But that proably won't help me speed up TCP.
Neville Franks Send private email
Tuesday, August 15, 2006
 
 
Samba is using natively UDP, which is as simple as possible within IP protocol. TCP has price in overhead for its advantage of behaving like pipe, just see any log from ethereal to see what's going on.

Another thing is saturation of movingwindow, TCP wont send blindly more data than set in its spec (u can regedit it and beef it up, but it will help only with small files) Also TCP's slow start algorithm sets back data transfer.

The best solution is to build own simple transfer protocol class which would use UDP, or google for it in case you sleept over network programming class at your uni :)

Its a bit complicated stuff at first, but when you get a grip it's simple as 3.14.. :) Docs for tcp and udp protocols are in RFC's and on wikipedia.

some useful links:
TCP - http://www.faqs.org/rfcs/rfc793.html
UDP - http://www.faqs.org/rfcs/rfc768.html
Beej's sockets guide - http://beej.us/guide/bgnet/

Cheers
Artur Sowinski Send private email
Tuesday, August 15, 2006
 
 
>The best solution is to build own simple transfer protocol class which would use UDP

But UDP doesn't gaurantee delivery, doesn't deliver packets in order etc.

Bed time for us folks down-under. Will check back early in the am.
Neville Franks Send private email
Tuesday, August 15, 2006
 
 
Of course UTP won't guantee delivery. When I spoke about own class implementing file transfer I assumed that things as checking integrity of transfered file (maybe extra crc'ing to be on a safe side) would be implemented in this class.
Artur Sowinski Send private email
Tuesday, August 15, 2006
 
 
UDP...
Artur Sowinski Send private email
Tuesday, August 15, 2006
 
 
If you are seeing a noticeable difference between UDP and TCP on a lan with I suspect it's a programming error.
Martin Send private email
Tuesday, August 15, 2006
 
 
At my uni we dissected tcp algorithm and had some theoretical excercises. On one we calculated maximum tcp speed based on it's default settings and specyfication to be something around 7Mbytes/sec, and that is independendent from underlying bandwidth.

Besides we're not speeking about lan but wlan, where dropped packet ratio can be anything form 0 to 100% depening on many stuff from hardware used to location of devices and walls between them
Artur Sowinski Send private email
Tuesday, August 15, 2006
 
 
Whenever I need to do something like this, I use ACE:

http://www.cs.wustl.edu/~schmidt/ACE.html

There's no way I could write better code on my own.
Jeff Mastry Send private email
Tuesday, August 15, 2006
 
 
You might also take a look at the RFC for FTP, http://www.faqs.org/rfcs/rfc959.html for some hints. It uses 2 channels to transfer data, one for command and control and the other for raw data transfer.

bcl
Brian C. Lane Send private email
Tuesday, August 15, 2006
 
 
Agree with Martin. Something else is going on.
son of parnas
Tuesday, August 15, 2006
 
 
This might be a dumb question...but are they connected via ethernet or is one of the PCs connected via wireless? I've found wireless to be a source of a large number of dropped/lossed packets in my own development.
Aspiring College Developer
Tuesday, August 15, 2006
 
 
We use to transfer large volumes of large files across the internet and found that Windows SMB takes twice as long as using HTTP (which is just using a TCP socket).

I think something is wrong in your setup or usage of TCP (buffering?) that is reponsible for the slow thoughput you observe.
Stefan Eissing Send private email
Tuesday, August 15, 2006
 
 
Aspiring, the original post says that the two PC's are on a wireless LAN.

Neville, can you connect the two machines with Ethernet?  If TCP vs. SMB performance is better with the wire, then the problem is in the wireless LAN.  TCP resends packets.  If your wireless LAN has interfering noise, bad line of sight, or other sources of dropouts, TCP will resend and resend, and restart if the wireless connection drops and reconnects.

Tuesday, August 15, 2006
 
 
>>The best solution is to build own simple transfer protocol class which would use UDP
>
>But UDP doesn't gaurantee delivery, doesn't deliver packets >in order etc.

Another alternative would be to check out something like Aspera Software's UDP-based file transfer protocol called FASP (http://www.asperasoft.com/).  The site also includes a lot of technical information explaining the advantages of their protocol over TCP-based ones.
Jared Send private email
Tuesday, August 15, 2006
 
 
Sorry, I read that as WAN not WLAN.
Aspiring College Developer
Tuesday, August 15, 2006
 
 
Turn off Nagle packet completion on TCP.

Tuesday, August 15, 2006
 
 
If you are using blocking sockets, it could explain -- unless you've been careful, using blocking sockets introduces a lot of latency which, with small buffer size, e.g. the default 8K, translates to low throughput.

Not less important, make sure you set the SOCKET buffer on both sides to a large value (e.g. 1MB or 4MB) -- this is done with the proper setsockopt() call.

SMB is slow; Much slower than plain HTTP or FTP. And for file transfer it uses TCP exclusively; It uses UDP for name resolution and few other minor functions.
Ori Berger
Tuesday, August 15, 2006
 
 
Thanks everyone.

The two PC's I'm using are connected via a Wireless LAN and are 8 feet from the WLAN Router. In fact one in connected to the Router via Ethernet and the other is a Notebook PC with 80211G WiFi. I can easily connect the Notebook via Ethernet but I'm comparing apples with apples, so how they are connected is not relevant with regard to my tests. ie. SMB and TCP will both suffer the same packet loss.

ACE is way too big and complex for my needs.

FASP could be interesting, but I don't want to used closed source code here.

I've not compared UDP and TCP. From my research UDP isn't used for this sort of thing.

I'm using WinXP on both PC's and haven't touched any TCP settings on either. And I send data in 64K chunks.

TransmitFile() and TransmitPackets() sound interesting.

The most interesting news is you folks saying that SMB is actually slower than TCP. I need to work out why I'm not seeing this. I think I need to write a simple test app and get a better understanding of what's going on.
Neville Franks Send private email
Tuesday, August 15, 2006
 
 
The phrase "SMB is slower than TCP" really doesn't make sense. They are two different layers of protocol - TCP for transport, and SMB for application (essentially). Any Win2K-and-later SMB transfers (both ends) are on TCP (I've analyzed dozens of traces of this kind of transfer). So SMB is the same speed as the TCP in that case, since they're both in use.

It is very possible to write file transfer stuff that's faster than SMB. FTP is an example (though I hate the two-connection, default-to-active design). There are two keys: 1) keep the data flowing to the kernel. On Windows, TransmitFile is easiest since it just says, "here's a file handle and a socket - send the data", and all the extra kernel mode transitions are avoided; 2) if you have to do individual sends, stream the data in one direction (don't send acknowledgement data from the receiver).

If you are targeting Windows on both ends, you could make things really easy and use the Ftp classes in the .NET framework and be done with it.

FWIW...

Donnie
soccerdad Send private email
Tuesday, August 15, 2006
 
 
Are you sending many small files, or one big file?

Anyway, I would want to get just the TCP functionality running first: a send and a receive process which call nothing but the WSA functions to send and receive data and to count and time the number of bytes transferred; use this to experiment with different buffer sizes; add logging to verify that the WSA functions are indeed always busy (i.e. that you're spending negligible time after completing one call before starting the next). Then, gradually add extra functionality like reading from file (to fill the send buffer with useful data) and writing to file (to empty the receive buffer to disk).

In the first version, avoid calling anything (including "new and so on, and especially including writing to the GUI, and verify that your CPU and disk are essentially idle).

Also, perfmon.exe might tell you interesting statistics, from the TCP stack and/or from your network card (number of dropped packets if any, and so on).
Christopher Wells Send private email
Tuesday, August 15, 2006
 
 
split your streams or files before sending.

Wednesday, August 16, 2006
 
 
"I'm using WinXP on both PC's and haven't touched any TCP settings on either. And I send data in 64K chunks."

I think that's probably 90% of your problem right there. A 64KB chunk, over a 54Mb/s link? That means you'd need to be sending a new chunk every 10ms or so to keep the pipe full. I kind of doubt you'll be able to do that effectively.

Try using larger chunks - maybe a megabyte each, or so. That'll drastically reduce the number of round-trips into the kernel.

Note that the same consideration applies to reading the files from disk in the first place - small I/O buffers will kill your read performance, too.

Alternatively, TransmitFile() sounds like it avoids the copying of data from the file read buffer to the network buffer, which should improve throughput a bit.
Mark Bessey Send private email
Wednesday, August 16, 2006
 
 
Another possibility is to get a good code profiler and find out just what your file transfer program is doing (i.e. spending its time on).
Artad Gobeski
Wednesday, August 16, 2006
 
 
>I think that's probably 90% of your problem right there. A 64KB chunk, over a 54Mb/s link? That means you'd need to be sending a new chunk every 10ms or so to keep the pipe full. I kind of doubt you'll be able to do that effectively.

I used 64K buffers because I read somewhere (can't find the reference right now) that that was the recommended maximum size to send. Seems like misinformation.

I've not had a chance to get back to this since my original post, but I'm definitely going to try TransmitFile(). I'll also put a simple test app together and do some more profiling.

Thanks again for everyones help.
Trying times! Send private email
Wednesday, August 16, 2006
 
 
As mentioned, and in my experience as well, you need to disable Nagle and use large buffers to get the best performance.
MBJ Send private email
Saturday, August 19, 2006
 
 
64K was the maximum TCP window size under NT 4.0 (and 9x?), but window scaling is available in Win2K+.  You'll probably want to set the option for a larger window as well as "send" (buffer) larger chunks.
Artad Gobeski
Saturday, August 19, 2006
 
 
I've made a some more progress on this and it seems I was wrong about TCP being a bottleneck.

I'm iterating a directory tree and updating files that have changed. When it hits some larger files (100MB+) it gets bogged down. If I update just one of these directories with these large files performance is fine. So I know think its related to disk i/o on the remote PC. It gets to a point where it can't keep up and I assume is spending time flushing its cache to disk, and this is where I'm seeing performance problems.

I'm only part way through this exercise though.
Neville Franks Send private email
Saturday, August 19, 2006
 
 
I tried various buffer sizes between 64K and 10MB and in my brief testa 256K gave the best result. But as I just wrote TCP isn't the problem after all.
Neville Franks Send private email
Saturday, August 19, 2006
 
 
> Directory trees

Try just doing it in memory and seeing what happens.

Directory trees are enormously slow on most OSes because they use brain dead data structures. It might better to not use large directories if you can.
son of parnas
Sunday, August 20, 2006
 
 
You never said how much of a difference there really is which makes it pretty difficult/impossible to suggest the right to
improve the throughput.

A 64 KiB buffer should be large enough almost all of the
time. Try dd and your disk for example which should be much
faster than 54 Mbit/s. I don't see any difference with buffer sizes beyond 32 KiB. Over a high latency connection, large buffer sizes might improve the performance otherwise 1 MiB of socket buffer is extreme overkill and just wastes RAM.

In any case, it's much more important to identify the bottleneck before jumping to conclusions. Try ping to identify the RTT and packet loss. If the former is high, larger buffers could help, if the latter is high, TCP might indeed have problems but writing a sane UDP-based file transfer protocol isn't as trivial as it may seem. Unfortunately, there isn't even a standard UDP-based protocol for this - TFTP definitely doesn't qualify.
Chris
Tuesday, August 22, 2006
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz