Division Error?

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

rhl - 24 Jul 2007 22:43 GMT

I just came across this weird division result with VB.NET.

2/3 = 0.66666666666666663

With VB6, I get

2/3 =0.666666666666667

Using the Windows Calculator results in
0.66666666666666666666666666666667

I'm sure I'm not the first one to notice this...what gives?

rhl

Reply to this Message

Scott M. - 24 Jul 2007 23:03 GMT

The differences are due to the math being done on different data types.

VB 6's, VB .NET's and Windows don't all have comprable data types.

>I just came across this weird division result with VB.NET.
>
[quoted text clipped - 10 lines]
>
> rhl

Reply to this Message

Patrice - 25 Jul 2007 17:43 GMT

Exact code ? I've got the same result than in VB6 with Debug.WriteLine(2 /
3)

Anyway, all those results are wrong and obviously you'll never be able to
display or even compute the correct result. More generally this is a well
know problem with "real" numbers.

You have several ways to workaround this depending on the problem you have
with this. You could use a better type (such as decimal) for increasing (and
possibly "guaranteed" precision), you could store this a a fraction etc...

--
Patrice

>I just came across this weird division result with VB.NET.
>
[quoted text clipped - 10 lines]
>
> rhl

Reply to this Message

Division Error?

Free Magazines