Xojo Conferences
MBS | Oct | 2019 | Cologne | DE |
Displayed Precision of Floats - What's Going On? (Real Studio network user group Mailinglist archive)
Back to the thread list
Previous thread: Text Tracks
Next thread: RB Crashes After Attempt to Edit Value in Variables Window
Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 06:10 (Wed, 29 May 2002 23:10:01 -0600) |
From: Steve Schacht |
Ok, I assign a value of _exactly_ 21.49 to a local variable of type Double.
(I actually type the literal value into the code window.) I insert a breakpoint to view the value of the local variable in the Variables window. If I size the Variables window so that the entire floating point value can be seen (so that the exponentiation symbol isn't there), it appears as 21.4899999999999984. If I resize the window smaller, it appears as 2.1490e+1. Similarly, if I assign the value 21.67 to the local float, it appears as 21.6700000000000017, or as 21.670e+1 when the window is made narrower. Why does the formatting change when the window is resized, and why is the precision off when exponentiation is not used? --- Steve Schacht <email address removed> --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 12:41 (Thu, 30 May 2002 06:41:33 -0500) |
From: CV |
Steve,
Just to clarify, when the Variable Window is resized down, 21.67 becomes 2.1670e+1, not 21.670e+1 as posted below. I'm sure the formatting changes in order to accomodate available space. With regard to the precision issue that you've raised, RealBasic will display floating point numbers to an apparent precision which is beyond the output precision available from the PowerPC Floating Point Standard (15-16 digits for doubles). You have noticed this in the Variables window and you can also see it in the returns of Format() if you designate placeholders for them. The additional digits displayed are part of the internal approximation of floating point numbers required because most floating point numbers cannot be represented exactly in binary. In other words, unless you have an interest in internal representations, display digits beyond 15-16 have no meaning as user output. Displaying these digits is in contrast to VB (VBA in Excel, for example, on the Mac), which will generally not display numbers beyond the 15-16 digits precision defined in the PPC standard for doubles. In my humble opinion(not widely shared), this is the ultimate cause of sometimes confusing display output in RB such as you have mentioned. However, RealBasic performs computations in compliance with the standard and you will notice that your two numbers are correct to the 15-16 digit precision as expected. Jack ----- Original Message ----- From: "Steve Schacht" <<email address removed>> To: "REALbasic NUG" <<email address removed>> Sent: Thursday, May 30, 2002 12:10 AM Subject: Displayed Precision of Floats - What's Going On? > Ok, I assign a value of _exactly_ 21.49 to a local variable of type Double. > (I actually type the literal value into the code window.) I insert a > breakpoint to view the value of the local variable in the Variables window. > If I size the Variables window so that the entire floating point value can > be seen (so that the exponentiation symbol isn't there), it appears as > 21.4899999999999984. If I resize the window smaller, it appears as > 2.1490e+1. Similarly, if I assign the value 21.67 to the local float, it > appears as 21.6700000000000017, or as 21.670e+1 when the window is made > narrower. Why does the formatting change when the window is resized, and > why is the precision off when exponentiation is not used? > > --- > Steve Schacht > <email address removed> > --- > Subscribe to the digest: > <mailto:<email address removed>> > Unsubscribe: > <mailto:<email address removed>> --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 21:42 (Thu, 30 May 2002 14:42:49 -0600) |
From: Steve Schacht |
On 5/30/02 5:41 AM, CV wrote:
> Steve, > > Just to clarify, when the Variable Window is resized down, 21.67 becomes > 2.1670e+1, not 21.670e+1 as posted below. Yes, that's correct. I goofed. > RealBasic performs > computations in compliance with the standard and you will notice that your > two numbers are correct to the 15-16 digit precision as expected. Thanks for the explanation. That helps me understand things a bit (no pun intended) better. Regards, --- Steve Schacht <email address removed> --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 14:55 (Thu, 30 May 2002 08:55:35 -0500) |
From: Thomas Reed |
>Ok, I assign a value of _exactly_ 21.49 to a local variable of type Double.
>[...] it appears as 21.4899999999999984. This is simply how things work on computers. Computers don't deal well with storing precise floating-point numbers. There is always inaccuracy, even before you've done any calculations. For this reason, you will have to treat such numbers a little differently. For example, if you do some calculations on a Double and want to see if the result is equal to 2.5, for example, you shouldn't do it like this: dim a,b,c as Double a = 1 b = 1.5 c = a + b if c = 2.5 then beep end if This code probably won't ever beep, because c will most likely be something like 2.50000000000001 or 2.499999999998. Instead, you need to decide on an acceptable margin for error and test that: a = 1 b = 1.5 c = a + b if abs(c-2.5) < 0.0001 then beep end if You'll run into the same problems whether you're writing code in RB, C++, or whatever. I imagine there are classes or modules out there that can do higher-precision handling of numbers, but I'm not sure what/where they are. -Thomas Personal web page: http://home.earthlink.net/~thomasareed/ My shareware: http://home.earthlink.net/~thomasareed/shareware/ REALbasic page: http://home.earthlink.net/~thomasareed/realbasic/ Pixel Pen web pub. guide: http://home.earthlink.net/~thomasareed/pixelpen/ Eat a live toad in the morning and nothing worse will happen to you for the rest of the day. --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 20:06 (Thu, 30 May 2002 15:06:11 -0400) |
From: Kevin Ballard |
On 5/30/02 9:55 AM, "Thomas Reed" <<email address removed>> wrote:
> This is simply how things work on computers. Computers don't deal well > with storing precise floating-point numbers. There is always inaccuracy, > even before you've done any calculations. For this reason, you will have > to treat such numbers a little differently. For example, if you do some > calculations on a Double and want to see if the result is equal to 2.5, > for example, you shouldn't do it like this: > > dim a,b,c as Double > > a = 1 > b = 1.5 > c = a + b > if c = 2.5 then > beep > end if > > This code probably won't ever beep, because c will most likely be > something like 2.50000000000001 or 2.499999999998. Instead, you need to > decide on an acceptable margin for error and test that: I just tested it with a couple of different values. It does beep. I guess RB automatically accounts for the acceptable margin of error with doubles? Or something like that? |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 20:14 (Thu, 30 May 2002 14:14:49 -0500 (CDT)) |
From: Craig A. Finseth |
> This is simply how things work on computers. Computers don't deal well
> with storing precise floating-point numbers. There is always inaccuracy, > even before you've done any calculations. For this reason, you will have > to treat such numbers a little differently. For example, if you do some > calculations on a Double and want to see if the result is equal to 2.5, > for example, you shouldn't do it like this: > > dim a,b,c as Double > > a = 1 > b = 1.5 > c = a + b > if c = 2.5 then > beep > end if > > This code probably won't ever beep, because c will most likely be > something like 2.50000000000001 or 2.499999999998. Instead, you need to > decide on an acceptable margin for error and test that: I just tested it with a couple of different values. It does beep. I guess RB automatically accounts for the acceptable margin of error with doubles? Or something like that? No. It's just that 0.5 is the one decimal fraction that works. Try: dim a,b as double dim i as integer a = 1/10 b = 0 for i = 1 to 10000 b = b + a next if b = 1000 then msgbox "ok" else msgbox "not ok" end Craig --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 31.05.02 19:09 (Fri, 31 May 2002 13:09:32 -0500) |
From: Thomas Reed |
>No. It's just that 0.5 is the one decimal fraction that works.
Ahh, I didn't know that. Sorry for giving such a number in the example, then! ;-) -Thomas Personal web page: http://home.earthlink.net/~thomasareed/ My shareware: http://home.earthlink.net/~thomasareed/shareware/ REALbasic page: http://home.earthlink.net/~thomasareed/realbasic/ Pixel Pen web pub. guide: http://home.earthlink.net/~thomasareed/pixelpen/ Always forgive your enemies, nothing annoys them so much. --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 20:17 (Thu, 30 May 2002 15:17:24 -0400) |
From: Caleb Strockbine |
On Thursday, May 30, 2002, at 03:06 PM, Kevin Ballard wrote:
> I just tested it with a couple of different values. It does beep. I > guess RB > automatically accounts for the acceptable margin of error with doubles? > Or > something like that? More likely, it's because 2.5 ( = 2 1/2) has an exact representation in binary numbers, so there's no rounding error. --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 20:23 (Thu, 30 May 2002 15:23:52 -0400) |
From: Kevin Ballard |
On 5/30/02 3:17 PM, "Caleb Strockbine" <<email address removed>> wrote:
>> I just tested it with a couple of different values. It does beep. I >> guess RB >> automatically accounts for the acceptable margin of error with doubles? >> Or >> something like that? > > More likely, it's because 2.5 ( = 2 1/2) has an exact representation in > binary numbers, so there's no rounding error. But, like I said, I tried more than just 2.5. Maybe I was just lucky. |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 22:48 (Thu, 30 May 2002 17:48:41 -0400) |
From: john roberts |
on 5/30/02 3:23 PM, Kevin Ballard at kevin@sb.org wrote:
>> More likely, it's because 2.5 ( = 2 1/2) has an exact representation in >> binary numbers, so there's no rounding error. > > But, like I said, I tried more than just 2.5. Maybe I was just lucky. No you are not wrong. They are not perfect yet but they have gotten much better. :) John --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: Displayed Precision of Floats - What's Going On? |
Date: 30.05.02 15:36 (Thu, 30 May 2002 09:36:21 -0500) |
From: Bob Delaney |
>
>This code probably won't ever beep, because c will most likely be >something like 2.50000000000001 or 2.499999999998. Instead, you need to >decide on an acceptable margin for error and test that: > > a = 1 > b = 1.5 > c = a + b > if abs(c-2.5) < 0.0001 then > beep > end if > >You'll run into the same problems whether you're writing code in RB, C++, >or whatever. I imagine there are classes or modules out there that can >do higher-precision handling of numbers, but I'm not sure what/where they are. > >-Thomas > My freeware plugins, MPCalc and Precision, handle up to 30,000 digits. http://homepage.mac.com/delaneyrm/ Bob --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |