Xojo Conferences
MBS | Oct | 2019 | Cologne | DE |
significant digits? (Real Studio network user group Mailinglist archive)
Back to the thread list
Previous thread: Primary Keys
Next thread: tab keys...
Re: REALbasic-NUG Digest #6108 - Mike Richardson | ||
significant digits? - David E. Garrett | ||
Re: significant digits? - john roberts |
significant digits? |
Date: 06.05.02 04:42 (Sun, 05 May 2002 21:42:43 -0600) |
From: David E. Garrett |
I am working on a project that needs me to round numbers to a specific
number of significant digits (usually 3). I envision something along the lines of: rounded = RoundToSD(unroundeddouble,howmanySD) To illustrate the rules of SD rounding (all to 3 SD): 2.345678 becomes 2.35 12.764 becomes 12.8 0.03453 becomes 0.0345 6.03456 becomes 6.03 9.3021 becomes 9.30 This is more complex than simple rounding to the nearest integer. I can mentally picture a method to do this, but it is so full of if this then that statements, I am sure I am thinking the hard way. Has anyone created such a method that handles this kind of rounding easily and elegantly? I am sure there is a clever and elegant mathematical approach one can take, but it escapes me. Any ideas or already done methods I can borrow? Thanks Dave ----------------------------------- David E. Garrett Teacher & Technology Coordinator Morinville Community High School Phone: (780) 939-6891 email: <email address removed> Fax: (780) 939-6896 http://www.mchs.gsacrd.ab.ca --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |
Re: significant digits? |
Date: 06.05.02 07:09 (Mon, 06 May 2002 02:09:55 -0400) |
From: john roberts |
on 5/5/02 11:42 PM, David E. Garrett at <email address removed> wrote:
> I am working on a project that needs me to round numbers to a specific > number of significant digits (usually 3). I envision something along the > lines of: > rounded = RoundToSD(unroundeddouble,howmanySD) > > To illustrate the rules of SD rounding (all to 3 SD): > 2.345678 becomes 2.35 > 12.764 becomes 12.8 > 0.03453 becomes 0.0345 > 6.03456 becomes 6.03 > 9.3021 becomes 9.30 > You might take a look at the format function and the scientific notation operator E. For example, let s be a string and x be a double. Using s = format(x,"-#.00") will put the string value of x into s with exactly 3 significant digits. Similarly, s = format(x,"-#.0000") will put the string value of x into s with exactly 5 significant digits. Both cases will be properly rounded BUT expressed in scientific notation. If this is OK, then you are done; however, judging from what you said above, you are not done. :) To guarantee keeping all significant digits you will have to shift the decimal yourself if the exponent is not zero. This will get you started; if you need some more drop me a note. Best regards, John Roberts mailto:<email address removed> --- Subscribe to the digest: <mailto:<email address removed>> Unsubscribe: <mailto:<email address removed>> |