Xojo Conferences
MBSOct2019CologneDE

Displayed Precision of Floats - What's Going On? (Real Studio network user group Mailinglist archive)

Back to the thread list
Previous thread: Text Tracks
Next thread: RB Crashes After Attempt to Edit Value in Variables Window


Re: Pigmy 2.0   -   Jan Erik Moström <
  Displayed Precision of Floats - What's Going On?   -   Steve Schacht
   Re: Displayed Precision of Floats - What's Going On?   -   CV
    Re: Displayed Precision of Floats - What's Going On?   -   Steve Schacht
   Re: Displayed Precision of Floats - What's Going On?   -   Thomas Reed
    Re: Displayed Precision of Floats - What's Going On?   -   Kevin Ballard
     Re: Displayed Precision of Floats - What's Going On?   -   Craig A. Finseth
      Re: Displayed Precision of Floats - What's Going On?   -   Thomas Reed
     Re: Displayed Precision of Floats - What's Going On?   -   Caleb Strockbine
      Re: Displayed Precision of Floats - What's Going On?   -   Kevin Ballard
       Re: Displayed Precision of Floats - What's Going On?   -   john roberts
   Re: Displayed Precision of Floats - What's Going On?   -   Bob Delaney

Displayed Precision of Floats - What's Going On?
Date: 30.05.02 06:10 (Wed, 29 May 2002 23:10:01 -0600)
From: Steve Schacht
Ok, I assign a value of _exactly_ 21.49 to a local variable of type Double.
(I actually type the literal value into the code window.) I insert a
breakpoint to view the value of the local variable in the Variables window.
If I size the Variables window so that the entire floating point value can
be seen (so that the exponentiation symbol isn't there), it appears as
21.4899999999999984. If I resize the window smaller, it appears as
2.1490e+1. Similarly, if I assign the value 21.67 to the local float, it
appears as 21.6700000000000017, or as 21.670e+1 when the window is made
narrower. Why does the formatting change when the window is resized, and
why is the precision off when exponentiation is not used?

---
Steve Schacht
<email address removed>

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 12:41 (Thu, 30 May 2002 06:41:33 -0500)
From: CV
Steve,

Just to clarify, when the Variable Window is resized down, 21.67 becomes
2.1670e+1, not 21.670e+1 as posted below.

I'm sure the formatting changes in order to accomodate available space.

With regard to the precision issue that you've raised, RealBasic will
display floating point numbers to an apparent precision which is beyond the
output precision available from the PowerPC Floating Point Standard (15-16
digits for doubles). You have noticed this in the Variables window and you
can also see it in the returns of Format() if you designate placeholders for
them. The additional digits displayed are part of the internal approximation
of floating point numbers required because most floating point numbers
cannot be represented exactly in binary. In other words, unless you have an
interest in internal representations, display digits beyond 15-16 have no
meaning as user output.

Displaying these digits is in contrast to VB (VBA in Excel, for example, on
the Mac), which will generally not display numbers beyond the 15-16 digits
precision defined in the PPC standard for doubles. In my humble opinion(not
widely shared), this is the ultimate cause of sometimes confusing display
output in RB such as you have mentioned. However, RealBasic performs
computations in compliance with the standard and you will notice that your
two numbers are correct to the 15-16 digit precision as expected.

Jack

----- Original Message -----
From: "Steve Schacht" <<email address removed>>
To: "REALbasic NUG" <<email address removed>>
Sent: Thursday, May 30, 2002 12:10 AM
Subject: Displayed Precision of Floats - What's Going On?

> Ok, I assign a value of _exactly_ 21.49 to a local variable of type
Double.
> (I actually type the literal value into the code window.) I insert a
> breakpoint to view the value of the local variable in the Variables
window.
> If I size the Variables window so that the entire floating point value can
> be seen (so that the exponentiation symbol isn't there), it appears as
> 21.4899999999999984. If I resize the window smaller, it appears as
> 2.1490e+1. Similarly, if I assign the value 21.67 to the local float, it
> appears as 21.6700000000000017, or as 21.670e+1 when the window is made
> narrower. Why does the formatting change when the window is resized, and
> why is the precision off when exponentiation is not used?
>
> ---
> Steve Schacht
> <email address removed>
> ---
> Subscribe to the digest:
> <mailto:<email address removed>>
> Unsubscribe:
> <mailto:<email address removed>>

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 21:42 (Thu, 30 May 2002 14:42:49 -0600)
From: Steve Schacht
On 5/30/02 5:41 AM, CV wrote:

> Steve,
>
> Just to clarify, when the Variable Window is resized down, 21.67 becomes
> 2.1670e+1, not 21.670e+1 as posted below.

Yes, that's correct. I goofed.

> RealBasic performs
> computations in compliance with the standard and you will notice that your
> two numbers are correct to the 15-16 digit precision as expected.

Thanks for the explanation. That helps me understand things a bit (no pun
intended) better.

Regards,

---
Steve Schacht
<email address removed>

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 14:55 (Thu, 30 May 2002 08:55:35 -0500)
From: Thomas Reed
>Ok, I assign a value of _exactly_ 21.49 to a local variable of type Double.
>[...] it appears as 21.4899999999999984.

This is simply how things work on computers. Computers don't deal well
with storing precise floating-point numbers. There is always inaccuracy,
even before you've done any calculations. For this reason, you will have
to treat such numbers a little differently. For example, if you do some
calculations on a Double and want to see if the result is equal to 2.5,
for example, you shouldn't do it like this:

dim a,b,c as Double

a = 1
b = 1.5
c = a + b
if c = 2.5 then
beep
end if

This code probably won't ever beep, because c will most likely be
something like 2.50000000000001 or 2.499999999998. Instead, you need to
decide on an acceptable margin for error and test that:

a = 1
b = 1.5
c = a + b
if abs(c-2.5) < 0.0001 then
beep
end if

You'll run into the same problems whether you're writing code in RB, C++,
or whatever. I imagine there are classes or modules out there that can
do higher-precision handling of numbers, but I'm not sure what/where they are.

-Thomas

Personal web page: http://home.earthlink.net/~thomasareed/
My shareware: http://home.earthlink.net/~thomasareed/shareware/
REALbasic page: http://home.earthlink.net/~thomasareed/realbasic/
Pixel Pen web pub. guide: http://home.earthlink.net/~thomasareed/pixelpen/

Eat a live toad in the morning and nothing worse will happen to you for
the rest of the day.

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 20:06 (Thu, 30 May 2002 15:06:11 -0400)
From: Kevin Ballard
On 5/30/02 9:55 AM, "Thomas Reed" <<email address removed>> wrote:

> This is simply how things work on computers. Computers don't deal well
> with storing precise floating-point numbers. There is always inaccuracy,
> even before you've done any calculations. For this reason, you will have
> to treat such numbers a little differently. For example, if you do some
> calculations on a Double and want to see if the result is equal to 2.5,
> for example, you shouldn't do it like this:
>
> dim a,b,c as Double
>
> a = 1
> b = 1.5
> c = a + b
> if c = 2.5 then
> beep
> end if
>
> This code probably won't ever beep, because c will most likely be
> something like 2.50000000000001 or 2.499999999998. Instead, you need to
> decide on an acceptable margin for error and test that:

I just tested it with a couple of different values. It does beep. I guess RB
automatically accounts for the acceptable margin of error with doubles? Or
something like that?

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 20:14 (Thu, 30 May 2002 14:14:49 -0500 (CDT))
From: Craig A. Finseth
> This is simply how things work on computers. Computers don't deal well
> with storing precise floating-point numbers. There is always inaccuracy,
> even before you've done any calculations. For this reason, you will have
> to treat such numbers a little differently. For example, if you do some
> calculations on a Double and want to see if the result is equal to 2.5,
> for example, you shouldn't do it like this:
>
> dim a,b,c as Double
>
> a = 1
> b = 1.5
> c = a + b
> if c = 2.5 then
> beep
> end if
>
> This code probably won't ever beep, because c will most likely be
> something like 2.50000000000001 or 2.499999999998. Instead, you need to
> decide on an acceptable margin for error and test that:

I just tested it with a couple of different values. It does beep. I guess RB
automatically accounts for the acceptable margin of error with doubles? Or
something like that?

No. It's just that 0.5 is the one decimal fraction that works.

Try:

dim a,b as double
dim i as integer

a = 1/10
b = 0
for i = 1 to 10000
b = b + a
next
if b = 1000 then
msgbox "ok"
else
msgbox "not ok"
end

Craig

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 31.05.02 19:09 (Fri, 31 May 2002 13:09:32 -0500)
From: Thomas Reed
>No. It's just that 0.5 is the one decimal fraction that works.

Ahh, I didn't know that. Sorry for giving such a number in the example,
then! ;-)

-Thomas

Personal web page: http://home.earthlink.net/~thomasareed/
My shareware: http://home.earthlink.net/~thomasareed/shareware/
REALbasic page: http://home.earthlink.net/~thomasareed/realbasic/
Pixel Pen web pub. guide: http://home.earthlink.net/~thomasareed/pixelpen/

Always forgive your enemies, nothing annoys them so much.

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 20:17 (Thu, 30 May 2002 15:17:24 -0400)
From: Caleb Strockbine
On Thursday, May 30, 2002, at 03:06 PM, Kevin Ballard wrote:

> I just tested it with a couple of different values. It does beep. I
> guess RB
> automatically accounts for the acceptable margin of error with doubles?
> Or
> something like that?

More likely, it's because 2.5 ( = 2 1/2) has an exact representation in
binary numbers, so there's no rounding error.

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 20:23 (Thu, 30 May 2002 15:23:52 -0400)
From: Kevin Ballard
On 5/30/02 3:17 PM, "Caleb Strockbine" <<email address removed>> wrote:

>> I just tested it with a couple of different values. It does beep. I
>> guess RB
>> automatically accounts for the acceptable margin of error with doubles?
>> Or
>> something like that?
>
> More likely, it's because 2.5 ( = 2 1/2) has an exact representation in
> binary numbers, so there's no rounding error.

But, like I said, I tried more than just 2.5. Maybe I was just lucky.

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 22:48 (Thu, 30 May 2002 17:48:41 -0400)
From: john roberts
on 5/30/02 3:23 PM, Kevin Ballard at kevin@sb.org wrote:

>> More likely, it's because 2.5 ( = 2 1/2) has an exact representation in
>> binary numbers, so there's no rounding error.
>
> But, like I said, I tried more than just 2.5. Maybe I was just lucky.

No you are not wrong.
They are not perfect yet but they have gotten much better. :)

John

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>

Re: Displayed Precision of Floats - What's Going On?
Date: 30.05.02 15:36 (Thu, 30 May 2002 09:36:21 -0500)
From: Bob Delaney
>
>This code probably won't ever beep, because c will most likely be
>something like 2.50000000000001 or 2.499999999998. Instead, you need to
>decide on an acceptable margin for error and test that:
>
> a = 1
> b = 1.5
> c = a + b
> if abs(c-2.5) < 0.0001 then
> beep
> end if
>
>You'll run into the same problems whether you're writing code in RB, C++,
>or whatever. I imagine there are classes or modules out there that can
>do higher-precision handling of numbers, but I'm not sure what/where they are.
>
>-Thomas
>

My freeware plugins, MPCalc and Precision, handle up to 30,000 digits.

http://homepage.mac.com/delaneyrm/

Bob

---
Subscribe to the digest:
<mailto:<email address removed>>
Unsubscribe:
<mailto:<email address removed>>