The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Using Java double for monetary amounts.

First, let me start by saying that I am well aware of all of the issues with floating point values for storing monetary amounts. Our Java system currently uses the double datatype throughout to store monetary amounts. It is too late to switch to BigDecimal so that simply isn't an option.

My main questions is this: Is the Java double data type capable of exactly representing all monetary amounts or are there certain values that simply can't be represented exactly? Can this be proven one way or another? Has anyone done the math on this before?
anon
Tuesday, May 23, 2006
 
 
Ok. Forget the question. I found an example which proves that there are at least some monetary amounts that can't be represented exactly. For example, 0.1 can't be represented exactly with a double.

I'm not actually sure whether or not this will impact us directly since these numbers ultimately get written into a database that is using fixed precision for the column.
anon
Tuesday, May 23, 2006
 
 
Not if you use dollars as your base unit. 1/10th is not representable as a binary fraction.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/html/_core_IEEE_Floating.2d.Point_Representation_and_Microsoft_Languages.asp
explains why.

Which means you can't represent $73.10 properly.
Katie Lucas
Tuesday, May 23, 2006
 
 
One thing to look at would be whether you ever use equality tests. This is a natural thing to do with monetary amounts (eg does sum_of_outgoings equal sum_of_incomings?), but an incorrect thing to do with floating point values (eg, in the .NET implementation of Double, 0.1+0.1+0.1 does not equal 0.3)
Larry Lard Send private email
Tuesday, May 23, 2006
 
 
Ofcourse it affects you.  If the tax-authority in your country of choice even thinks you are evading taxes you'll have to bend over, tuck your toes and breath deeply.
Dave
Tuesday, May 23, 2006
 
 
The only fractional amounts that can be evenly represented are $0.25, $0.50, and $0.75. All other fractions will be off by at least a little bit.

Those errors will tend to cancel each other out. Though, in the long run, you will be fleecing a tiny bit of money from your customers. I wrote a little but of test code (included at the end of this message) which demonstracts that for every quadrillion dollars of transactions processed by your system, you will be scamming three cents from your customers.

So, whether you choose to fix the bug is a business decision regarding whether the company thinks that kind of inaccuracy would be a problem. For the world's largest financial providers, it could possibly add up in the long run (though, with 64 bits of precision, it's hard to see why anyone would *really* care all that much).

<code language="Java">
package net.benjismith.junk;

import java.math.BigDecimal;

public class DecimalTest {
 
  public static void main (String[] args) {
    BigDecimal totalError = new BigDecimal(0);
    for (int pennies = 0; pennies <= 100; pennies++) {
      BigDecimal dollars = new BigDecimal(pennies).divide(new BigDecimal(100));
      double wrong = dollars.doubleValue();
      BigDecimal error = new BigDecimal(wrong).subtract(dollars);
      System.out.println(pennies + " pennies, error = " + error);
      totalError = totalError.add(error);
    }
    BigDecimal quadrillion = new BigDecimal("1000000000000000");
    System.out.println("error per $1000: " + totalError.multiply(quadrillion));
  }

}
</code>

Now, if you re-run that same code above, but you use floats instead of doubles, the problem is MUCH MUCH more pronounced. With only 32 bits of precision, your customers would lost three cents per ten million dollars in transactions. Since most banks easily perform hundreds of billions of dollars in transactions per year, that kind of error would definitely be unacceptable.
BenjiSmith Send private email
Tuesday, May 23, 2006
 
 
The best thing to do is to create a money class, it works wonders. Using even a BigDecimal for money is complicated, so wrapping up this complication in a well tested money class is a very good benefit. I have been using this for a while now and it's really handy.

Check out: http://www.martinfowler.com/eaaCatalog/money.html

While this is just a vague description, the actual book gives enough code (in Java) to create your own, there is at least one bug in the books code but it's not hard to figure out what's wrong.

Also since I never work with foreign currency I left the currency part out which simplifies database access, but if I ever needed it having everything in a class already would make adding it much easier.
Justin Kolb Send private email
Tuesday, May 23, 2006
 
 
Oops. In the code that I posted, that last println should say "error per $quadrillion", not "error per $1000".
BenjiSmith Send private email
Tuesday, May 23, 2006
 
 
But there is a difference between "not being able to represent 0.1 properly" and being impacted by it. For example, the following code displays the correct result.

double value = 0.1;
System.out.println("" + value);

The value displayed is "0.1". Not something like "0.09999...". So Java is doing an internal conversion using some sort of error algorithm to determine that 0.1 is close enough to the actual floating point binary value. The same is true of the database. The JDBC driver is smart enough to know that the value to be stored in the database is 0.1.

The problem occurs when the floating point error is large enough that the internal conversion routines no longer consider the two values to be equal and displays the wrong value. But can it be proven that this will never happen if I am careful to round after computations?

Also, I realize that for very large numbers the cents value of a number can easily be lost due to the number of bits of precision. But we simply don't deal in numbers that large.
anon
Tuesday, May 23, 2006
 
 
The problem is that people with money are very picky, if you lose a couple of cents here and there they tend to get irate.

Money is an exact precision number better represented by a long or a BigDecimal number. Double and Floats are good for scientific calculations.

You don't want conversions behind the scenes when dealing with money, you need to know exactly what's going on and when. You need to know that 0.01 cents is exactly 0.01 cents and not really 0.00987 cents behind the scenes.

Yes having to convert to not use floats/doubles will be a pain in the ass, but I'm pretty sure that not doing so could adversely affect your job.
Justin Kolb Send private email
Tuesday, May 23, 2006
 
 
"For example, the following code displays the correct result."

anon, you have a subtle misunderstanding of what's actually going on. The reason that code prints out "0.1" is because the Double.toString() method performs a rounding operation before it creates the String object.

But the internal representation is still inaccurate. There is no implicit conversion going on behind the scenes. The same is true of the database. It will not quietly convert values for you in the background, particularly since there is *no*possible*way* to accurately represent 0.1 as a floating point number.

So, you may decide not to fix the bug (and make no mistake, it's definitely a bug) because you believe the financial impact will be minimal. And I think that could be a valid choice (depending on how difficult it would be to fix the bug by using BigDecimal). But whatever you do, don't leave the bug unfixed because you believe that Java is just magically fixing the bug for you. Cuz that's definitely not the case.
BenjiSmith Send private email
Tuesday, May 23, 2006
 
 
Justin, we are talking about hundreds of thousands of lines of code. I didn't write this system. I inherited it. We can't change to using another data type. But we can do damage control and that's what I'm trying to evaluate.

Thanks for all of the help everyone. It sounds like I need to do some more research into how Java performs the conversion between the internal floating point representation and the values displayed to the user and written to the database. It is at these input and output points that we need to be correct.
anon
Tuesday, May 23, 2006
 
 
Benji...

"anon, you have a subtle misunderstanding of what's actually going on. The reason that code prints out "0.1" is because the Double.toString() method performs a rounding operation before it creates the String object."

Yes, I am confused. This is why I asked the question in the first place. If the Double.toString() method can perform this rounding to make it "exact" then surely I could as well. But mathematically that doesn't make sense. So the Double.ToString() method is technically not "rounding" per se. It must be using some sort of error analysis to come up with what it thinks the result should be. Some could even call this a bug if they desired all of the trailing decimal places to be present in the output. So somewhere this is a disconnect in my understanding of what the Double.ToString() method is doing.

"But the internal representation is still inaccurate. There is no implicit conversion going on behind the scenes. The same is true of the database. It will not quietly convert values for you in the background, particularly since there is *no*possible*way* to accurately represent 0.1 as a floating point number."

But the database is not storing the value as a floating point number. I believe that it is storing it as a fixed point value. So some conversion must be taking place and again, it appears to be correct.

Yes, this is a bug and I understand that the internal representation is not correct. But is there a way that I can remove the impact of these types of floating point errors? That's what I'm really not sure about.
anon
Tuesday, May 23, 2006
 
 
Ouch, once again the JOS forum leads to a good question for interviews -- "What data type would you use to represent a monetary amount?"
SomeBody Send private email
Tuesday, May 23, 2006
 
 
It's not doing any error analysis.  Just read the javadoc of Double.toString(double d).  It's certainly a mouthfull.

Anyway, I had (sortof) your problem in a webapp once.  Eventually with the team we decided the doubles would be removed and we would use a class instead.  I did that conversion IIRC in just under 5 days.  It wasn't a full solution, but good enough.  I estimate the app had about 8 to 20 manyears at that point.  Now, most of that had nothing to do with calculations, so this may not apply to you. 

But are you sure it would be undoable to just go in, rip out 99% of the low-hanging fruit that represents your problem and replace it with something a whole lot better?
Dave
Tuesday, May 23, 2006
 
 
Thanks Dave. Reading the Double.ToString() javadoc does help me understand it better.

As I said before, switching to another data type really isn't an option. You have to understand that the same guy who made the initial decision to use doubles to represent monetary amounts is the same guy who created everything else. In other words, the code is a real mess. ;)
anon
Tuesday, May 23, 2006
 
 
"Yes, I am confused. This is why I asked the question in the first place."

Sorry, I didn't mean to sound critical. It's a good question. My opinion is: don't fix the bug. But also dont' misunerstand the reasons for not fixing the bug. Acknowledge that the bug exists, but that there's about a better than one-in-a-trillion chance that it will cause financial problems for you or your customers (if you were using 32-bit floats, then you'd have a problem, and you'd definitely need to fix it).

"So the Double.ToString() method is technically not 'rounding' per se."

Aha. Good point. According to the Javadoc, "There must be at least one digit to represent the fractional part, and beyond that as many, but only as many, more digits as are needed to uniquely distinguish the argument value from adjacent values of type double. That is, suppose that x is the exact mathematical value represented by the decimal representation produced by this method for a finite nonzero argument d. Then d must be the double value nearest to x; or if two double values are equally close to x, then d must be one of them and the least significant bit of the significand of d must be 0."

So, technically, they can output whatever String they want, as long as there no other possible double values closer to the actual number represented in the string. Very tricky. It's not rounding, but it's kind of like "implied rounding".

Incidentally, you mentioned that the database uses fixed-point numbers, and that you should be able to safely represent decimal values in the DB because of that. Unfortunately, that's also a false assumption. Fixed-point numbers are no more accurate than floating-point numbers. You still can't represent 0.1 accurately in fixed-point format. A floating point number just takes a few extra bits to encode the *precision* of the number. More details here:

http://en.wikipedia.org/wiki/Fixed-point_arithmetic

If your database is storing the values as "decimal" values, then you're in better shape. But fixed-point buys you nothing (except the ability to represent much much smaller and much much bigger values (like 1.0 x 10^200, which would be impossible to represent with a 64-bit fixed-point number, but which can easily be represented with a floating-point number of the same size)).
BenjiSmith Send private email
Tuesday, May 23, 2006
 
 
Here is a simple class that shows how easy it is to lose/gain pennies. Notice after I added 4 float values together I've gained 4 cents on the total. Also I attemted to add in some rounding and then truncation code to convert the floats into ints temporarily that didn't work either. There is nothing that will work (PERIOD). The only option is to use a long or a BigDecimal as your money value.

package test;

import java.math.BigDecimal;
import java.text.DecimalFormat;
import java.text.NumberFormat;

public class FloatMoneyTest {
    private static final NumberFormat numbers1 = new DecimalFormat("0.000000000000");
    private static final NumberFormat numbers2 = new DecimalFormat("0.00");

    public static void main(String[] args) {
        BigDecimal fromDatabase1 = new BigDecimal("123.86");
        BigDecimal fromDatabase2 = new BigDecimal("123.06");
        BigDecimal fromDatabase3 = new BigDecimal("123.07");
        BigDecimal fromDatabase4 = new BigDecimal("123.03");
       
        System.out.println("Amounts in database:");
        System.out.println(numbers1.format(fromDatabase1));
        System.out.println(numbers1.format(fromDatabase2));
        System.out.println(numbers1.format(fromDatabase3));
        System.out.println(numbers1.format(fromDatabase4));
        System.out.println();
       
        float inProgram1 = fromDatabase1.floatValue();
        float inProgram2 = fromDatabase2.floatValue();
        float inProgram3 = fromDatabase3.floatValue();
        float inProgram4 = fromDatabase3.floatValue();
       
        System.out.println("Amounts in program:");
        System.out.println(numbers1.format(inProgram1));
        System.out.println(numbers1.format(inProgram2));
        System.out.println(numbers1.format(inProgram3));
        System.out.println(numbers1.format(inProgram3));
        System.out.println();
       
        float addedInProgram1 = inProgram1 + inProgram2 + inProgram3 + inProgram4;
        BigDecimal addedInProgram2 = fromDatabase1.add(fromDatabase2).add(fromDatabase3).add(fromDatabase4);
       
        System.out.println("Amounts added as floats:");
        System.out.println(numbers1.format(addedInProgram1));
        System.out.println(numbers2.format(addedInProgram1));
        System.out.println();
       
        System.out.println("Amounts added as BigDecimals:");
        System.out.println(numbers1.format(addedInProgram2));
        System.out.println(numbers2.format(addedInProgram2));
        System.out.println();
       
        System.out.println("Proposed fix:");
        int intValue1 = Math.round(inProgram1 * 100.0f);
        int intValue2 = Math.round(inProgram2 * 100.0f);
        int intValue3 = Math.round(inProgram3 * 100.0f);
        int intValue4 = Math.round(inProgram4 * 100.0f);
        int addedIntValues = intValue1 + intValue2 + intValue3 + intValue4;

        System.out.println(intValue1);
        System.out.println(intValue2);
        System.out.println(intValue3);
        System.out.println(intValue4);

        System.out.println("Added ints:");
        System.out.println(addedIntValues);
        float converToFloat = addedIntValues / 100.0f;
        System.out.println("Converted to float:");
        System.out.println(numbers1.format(converToFloat));
        System.out.println(numbers2.format(converToFloat));
        System.out.println();
       
        System.out.println("Proposed fix 2:");
        intValue1 = (int)(inProgram1 * 100.0f);
        intValue2 = (int)(inProgram2 * 100.0f);
        intValue3 = (int)(inProgram3 * 100.0f);
        intValue4 = (int)(inProgram4 * 100.0f);
        addedIntValues = intValue1 + intValue2 + intValue3 + intValue4;

        System.out.println(intValue1);
        System.out.println(intValue2);
        System.out.println(intValue3);
        System.out.println(intValue4);

        System.out.println("Added ints:");
        System.out.println(addedIntValues);
        converToFloat = addedIntValues / 100.0f;
        System.out.println("Converted to float:");
        System.out.println(numbers1.format(converToFloat));
        System.out.println(numbers2.format(converToFloat));
        System.out.println();
    }
}
Justin Kolb Send private email
Tuesday, May 23, 2006
 
 
LOL there was a bug in the program!
Justin Kolb Send private email
Tuesday, May 23, 2006
 
 
So I guess you are OK then, and I'm a dumbass =).
Justin Kolb Send private email
Tuesday, May 23, 2006
 
 
I do know that if you do any tax calculations or any type of multiplication your rate of errors starts increasing.
Justin Kolb Send private email
Tuesday, May 23, 2006
 
 
Just use a larger integer type (ie, long) Keep the value in pennies (or whatever the smallest unit of currency is if you're not in the US.) If you need more, use an overflow field. Problem solved.
Dave
Tuesday, May 23, 2006
 
 
If you need to switch to a different datatype, int or long are definitely the wrong answer. If you code it with the assumption that you only need to support penny-accurate calculations, then someone will come along later and change those assumptions.

If you think tenth-of-a-penny is good enough, you will eventually be proven wrong by a change in requirements.

Since the OP isn't planning on changing the datatype, it's kind of a moot point. But if he was willing to change the datatype, the correct answer is BigDecimal, not int or long.
BenjiSmith Send private email
Tuesday, May 23, 2006
 
 
One solution is to write a money class implemented using a 64-bit int with 4 implied decimal places. One of the companies I worked for back in the day did the same using a 32-bit unsigned int (money stored as pennies) and never had any problems with rounding errors. Of course, the amounts they processed were in the tens of thousands of dollars, max.
MBJ Send private email
Tuesday, May 23, 2006
 
 
Yeah, that's one solution.

The *wrong* solution.

Like I said before, if he's going to switch datatypes, a completely robust decimal class already exists. It doesn't require you to commit to a particular level of precision (you can get more later, if you need it) and it doesn't require you to accept some arbitrarily low ceiling.

There's no need to reinvent this particular wheel.

Do not use an int. Do not use a long.

If you must stick with an existing (broken) implementation, then so be it. But if you're going to fix the problem, at least use the correct solution.
BenjiSmith Send private email
Tuesday, May 23, 2006
 
 
Benji,

I work for a company processing 100's of millions a year, and we use longs (actually, 2 32-bit longs, one being an overflow  counter.) All transactions are represented in pennies. We do not process fractional pennies, though there is no reason the system couldn't be extended that way.

You need to define the precision during the requirements phase. Do you want 2 decimal places? 4? To say it's not part of the requirements is idiotic.
Joe Send private email
Tuesday, May 23, 2006
 
 
Oh, and this assumes you are STORING the amounts in some type of database. Few databases are going to understand a "BigDecimal", so it will either be represented as a long or a float (with limited precision.)
Joe Send private email
Tuesday, May 23, 2006
 
 
"Just use a larger integer type (ie, long) Keep the value in pennies (or whatever the smallest unit of currency is if you're not in the US.) If you need more, use an overflow field. Problem solved."

Yes, that's what we used to do in the "old days". Things really should be better now. If I were going to switch the data type I would go with BigDecimal in Java. .NET has the decimal datatype which is a huge improvement over Java. But as Benji pointed out I don't plan on changing the data type anyway. :)

Thanks again everyone. This discussion has been very enlightening.
anon
Wednesday, May 24, 2006
 
 
"I work for a company processing 100's of millions a year, and we use longs (actually, 2 32-bit longs, one being an overflow  counter.) All transactions are represented in pennies. We do not process fractional pennies, though there is no reason the system couldn't be extended that way."

When you say that the system could be "extended" to handle fractional pennies, would that be difficult? Could you make code changes in just one place, or would they be spread out all over the source? I'm sure there's a good reason that your system is coded with longs rather than decimals (possibly because your system predates the availablility of a de-facto decimal class?)

"You need to define the precision during the requirements phase. Do you want 2 decimal places? 4? To say it's not part of the requirements is idiotic."

I've never worked on an application where the requirements were set in stone and then never changed again. Usually, requirements evolve in subtle ways over the course of many years. Hopefully, at the beginning of the implementation process, we make wise choices so that we don't paint ourselves into a corner when the requirements inevitably change. Using a decimal class that allows us to use as much precision as we need just seems like a good design decision.

I just can't quite fathom why someone would choose to use an integral datatype or a floating-point datatype to represent a decimal problem domain, when there are already perfectly good decimal datatypes available in nearly every programming language and database environment. The only justifyable reason seems to be if you're working with a legacy system. That's all I'm saying.

"Oh, and this assumes you are STORING the amounts in some type of database. Few databases are going to understand a 'BigDecimal', so it will either be represented as a long or a float (with limited precision.)

MySQL, Postgres, and MS SQL all have DECIMAL data types. Oracle's NUMBER type uses decimal-digit precision rather than binary-digit precision.

So which databases were you referring to where currency has to be stored as a long or a float?
BenjiSmith Send private email
Wednesday, May 24, 2006
 
 
An IEEE 754 double precision number has a mantissa of 53 bits. So for an amount of 1 billion dollars (30 bit), your inaccuracy is 2^-23, or about 0.00000012 dollar. With a maximum inaccuracy of 0.005 dollar, you can add about 40000 of those double precision numbers (containing less than 1 billion) before you start losing pennies. Multiplication and division are more tricky.

You can convert any double back to its minumum inaccuracy this way:

multiply by 100
round it
divide by 100

Do this early and often, and you can keep your inaccuracies from growing into real pennies.

Comparison has to be done with the minimum precision, like this:

if ((a + 0.0049999999 > b) && (a - 0.0049999999 < b)) {
  // consider a and b equal
}

If you can use pennies as the unit, the problem becomes much much smaller. Floating point numbers can represent integers up to the size of the mantissa exactly. That is why the rounding works. When you use pennies as your unit, you only have to round the number after any division or multiplication by a fraction, etc.

There is still a catch when comparing numbers. There are two zeroes in the floating point world: positive and negative zero. In other words: +0.0 != -0.0

See http://en.wikipedia.org/wiki/IEEE_754 for more information and links about floating point numbers.
O.Hovers
Tuesday, June 06, 2006
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz