The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Never Fast Enough

So, I have this Java program that reads every byte of a 3GB file, calls a static method that maps the byte from its original form (EBCDIC) to something useful (ASCII) in a large 'switch', then writes it to a new file. It is disappointingly slow, 3-5 hours on a HPUX 9000 server.

I theorize it's because of the method call overhead (well, its being called 3 billion times for Pete's sake!), because the read/write operation itself only takes 5-10 minutes (and yes, it uses BufferedReader etc.).

I would like to speed this up, would inlining the method call significantly help? Any other bright ideas? In case its not obvious, I am NOT a Java expert, by the way.
NetFreak Send private email
Monday, September 26, 2005
Try using a lookup table and ditch the method call.
Monday, September 26, 2005
Just last week I wrote a C program that converts 200MB files it actually needs to parse (simple parsing, but significantly more than ebcdic->ascii). It uses mmap() for both in and out to speed things up.

Running on a Pentium 3 1.4Ghz, 512MB of Ram it takes ~20 secs for each 200MB file if the file is not cached and < 5 if it is (meaning, IO overhead is ~15 secs).

If you're forced to stay in Java, look at memory mapping primitives (I think they were introduced in JDK 1.4), they might help.

Other than that, find a unix station and type "man dd". It has a feature that converts ascii<->ebcdic; It will work quickly _and_ save you the coding time. Failing that, install cygwin's or gnuwin32's "dd" version, but they are both slow compared to any decent Unix I've used.
Ori Berger
Monday, September 26, 2005
I assume you have profiled the code to see where the time is being spent?

In any case you are probably better off parallelizing the operations by dividing the file up into sections that are processed separately. The results are then merged together.
son of parnas
Monday, September 26, 2005
'dd' doesn't handle packed fields, and packed fields also make splitting files up problematic.

If you are into the whole Jurassic data formats thing, you'll find that EBCDIC is simple (actually FTP usually deals with it); packed fields are the big complication.

Anyways, thanks!
NetFreak Send private email
Monday, September 26, 2005
The method is more than likely already being inlined. I'd first try the server VM (-server flag to the java executable). Then have a look at your buffer sizes. Are you using blocking or non-blocking io?
Monday, September 26, 2005
This sort of problem is deceptively hard in Java.  Most of the overhead in this type of program is caused by object creation and destruction.  Here's a couple things I'd try:

1)  Read and write the files as binary data.
2)  Make sure the optimizer inlines your conversion code

This sounds like a great project for a contest!  Who can make the fastest converter?  Who can make the smallest?  Who can do it with the least lines of code (this should result in some code that violates Java coding guidelines)?

Of course, you'll have to test the entries ... we can't be uploading and downloading 3GB files.
Steve Moyer Send private email
Monday, September 26, 2005
Obvious problem: the switch statement, you need a simple table which is so easy. Simple EBCDIC to ASCII? You should be doing 3GB in well under a minute including read/write, in fact the only hold-up is the file I/O.
Anon and anon
Monday, September 26, 2005
First of all, ditch the BufferedReader/BufferedWriter classes. They're not appropriate for operating on a file this large.

Look into the FileChannel and ByteBuffer classes in the java.nio package. Allocate the ByteBuffer with the allocateDirect() method, which will provide you with memory-mapped IO into your input file. That should speed things up significantly.

Then, as someone else suggested, write the conversion function as a lookup table.

And let us know how it all works out.
BenjiSmith Send private email
Tuesday, September 27, 2005
Why do a byte at a time?  Why not do a whole int?

This would cut your function call overhead by four.  Trivial to code, too.  See if it helps.

Nothing like the blind leading the blind ;-)
new nick, new rep
Tuesday, September 27, 2005
Read, convert and write in big chunks, not byte per byte. That alone will give you a drastic speed up. Do conversion inline, without method calls. This will also give you much speed up. Finally, ditch Java and make this in a language that compiles into native code (yes, I know about "JIT compilation", it still is slower than native code and will be). That will give you the last speed up, and if you are still not satisfied, then you'll have to optimize the hell out of it manually. But I suspect it won't come to that.
Tuesday, September 27, 2005
I've written a Java EBCDIC<->ASCII converter not using static lookup tables (we've to work with different understandings of what EBCDIC means (no, don't ask)) and it needs about 40secs for a 170MB file, which means it'll take about 12minutes for 3GB. Sadly I've done it for the company which means I won't be able to post the source here, but if you got any detailed questions, I might be able to help you...

The basic trick is to read blocks of data into a byte array and then convert the whole block instead of reading it byte by byte...
Helge Richter Send private email
Tuesday, September 27, 2005
If you followed the various advice, you can try speeding it up by building a lookup table of character pairs.

Basically, you create 32,768 pairs of all possible EBCDIC character pairs (AA, AB etc.) and sort them by the indices of their ASCII equivalents.

Treat a character as a 16-bit integer and use that as an index into the EDCDIC array. The EBCDIC array will fit in 64 KB of cache.

The speed of conversion will be the time needed to read the file from disk (in 10 MB chunks for ex.), plus a few seconds for the conversion, plus the time needed to write all 10 MB chunks back to disk.

If, all in all, it takes more than two minutes to convert the file, there is either something wrong with your algorithm, or with Java :)
Frank de Groot Send private email
Tuesday, September 27, 2005
Using on-the-fly created temporary lookup tables seems to be a lost art. Some C-code:

  unsigned char Asc_Of_Ebc[256];
  int i;

  for (i = 0; i < 256; i++)
      Asc_Of_Ebc[i] = Existing_Ascii_Of_Ebcdic(i);

And some C-pseudocode for the file conversion:

  unsigned char *Buffer;
  uint32 Read, i;

  Buffer = Allocate(1024*1024);

Don't use just 1024; make your harddisk go a silent TICK-TICK-TICK instead of a loud TRRRRRRRRRRRRR by using a bigger buffer. We are dealing with 3GB of mixed read/write operations; you won't water your plants with a thimble (and 3GB is no bonsai).
  while ((Read = FileRead(Infile, Buffer, 1024*1024)) != 0)
      for (i = 0; i < Read; i++)
          Buffer[i] = Asc_Of_Ebc[ Buffer[i] ];
      FileWrite(Outfile, Buffer, Read);
Tuesday, September 27, 2005
Syncsort makes a program called FilePort, which will blast EBCDIC files over to ASCII. It understands COBOL copybooks, or so the documentation says. If we had to convert such files all the time, we'd certainly get it.
George Jansen Send private email
Tuesday, September 27, 2005
Thanks for all these great ideas!

Helge Richter and others: I oversimplified the original description slightly; it actually reads in record-sized chunks, about 300 bytes at a shot, and in any case, I am using BufferedReader/Writer.

I sincerely doubt a commercial product would work because besides packed fields, which I agree some products CAN handle, this particular file uses a relatively complex series of offsets and length pointers to chain everything together. You have to look at 40 different COBOL-style copybooks and relate them together (eg. whether or not copybook X applies depends on the segment size in copybook Y, and so on).

Its definetely NOT rocket science, but hard to see how a canned product could figure this out from copybooks.

Anyways, thanks again everyone; onwards and upwards!
NetFreak Send private email
Tuesday, September 27, 2005
don't waste time doing an int table instead of a char table. Once you do the conversion with a 256 entry table, outbuf[x]=ascii_table[inbuf[x]], your only bottleneck will be I/O. The conversion of a MB buffer in memory (in C++) would be less than 10 milliseconds (1/100th of a second) on an average PC. I am not familiar with the Java issues. It is so funny every time I hear people putting up with hours to do something so simple and fast!
Anon and anon
Tuesday, September 27, 2005
What is the size of the Buffer for your BufferedReader/BufferedWriter?  Considering the size of the file I would buess a buffer ~1meg in size would work well.

Make sure you don't call flush on the writer, since that would completely screw up the use of a buffer.
Chris Mercer Send private email
Wednesday, September 28, 2005
> Try using a lookup table and ditch the method call.

It's not the method call, but the switch statement instead of  the table.

Also, I would really recommend Perl for this if you are familiar with it.

A thread on the perils of a Java only approach.
Getting annoyed now
Thursday, September 29, 2005
I tried removing the method call AND the switch and replacing it with writing the same number of constant characters to the output file. Time dropped by about half, which is great, but it still leaves it in the multi-hour category.

I think it spends a lot of time iterating thru an array that tells it how big each field is and how to translate it. I simplified this out of the original post. It is probably scanning an average of 125 elements in an array once per record which is about 8.5 million times. I intend to benchmark how long this takes by itself, next.

I doubt buffer sizes will help because I tested simply reading and writing with no calls or anything and it took a few minutes total, which is what I'm shooting for.

Thanks again to all who posted.
NetFreak Send private email
Thursday, September 29, 2005
> I think it spends a lot of time iterating

You don't think. If you were thinking you would profile the code and figure out what the heck is really going on instead of fumbling around like a teenager trying to unhook their first bra.
son of parnas
Thursday, September 29, 2005
Like son of parnas says, you really should be profiling this thing.

Here's a pretty good free profiler that plugs into eclipse...
BenjiSmith Send private email
Friday, September 30, 2005
You should use object oriented methods. Refactor your switch into classes. Create a set of 256 functions, each which translates a different character. Then, when you get a character, call a factory to instantiate one of those methods and place it in a vector. Then iterate through the vector using your base class, calling the virtual convert method on each element.
Art Wilkins
Sunday, October 02, 2005
Thank you, SOP; when it comes to Java, I admit that your characterization is not entirely unfair.
NetFreak Send private email
Thursday, October 06, 2005

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz