Re: A86: calculation error
[Prev][Next][Index][Thread]
Re: A86: calculation error
At 21:05 1998-10-30 -0500, you wrote:
>
>has anyone else noticed that when doing math calculations, sometimes the
>answer will be 4.000000000001 or 3.999999999999 instead of 4 (4 is just an
>example)? in other words, it's off by the smallest value possible. any ideas
>why?
I don't see what this has to do with assembly, but the answer is
pretty easy. First you can try calculating (1/3+1/3+1/3)-1 on any
non-symbolic calculator, and you will get the anwer -1E-14 or some
such instead of 0. The reason is that a computer (and calculator)
uses a limited storage space for numbers (unlimited memory is
very hard to find :) ), and thus rounding errors occur. In the
case above you add .333+.333+.333 = .999, and then -1 = -0.001.
--
Real name: Jimmy Mårdell
Email: mailto:yarin@acc.umu.se
Homepage: http://www.acc.umu.se/~yarin/
Icarus Productions homepage: http://icarus.ganymed.org/
References: