Re: Another hyperspace bug found --> random and Neural Net

On Wed, Jun 18, 2008 at 9:53 PM, Bill Yerazunis <[email protected]> wrote:
> (note- several of the classifiers have slightly different results
> _just changing from -O0 to -O3... this is very interesting, bordering
> on suspicious, so if anyone else sees this, please let me know.  Note that

Yeah. Seen that. Simply put, it's the can of worms opened up by the
combo of the varying allowances (due to optimization settings) for the
compiler and processor in handling floating point calculations and
conversions (from float to double and vice versa, conversion to/from
long, these all happen in a lot of places in crm114. I've found it
helps the numbers quite a bit when one uses double for internal
statistics calculations where applicable, ONLY doing conversion
to/from float and long(=int32_t) when accessing persistent storage
content (CSS files). Implicit and explicit casts are where compilers
are allowed to play smart.

Outside the GCC/Linux realm, there still more worms waiting for ya.
Microsoft VC can be set to optimize 'fast math', something akin to
gcc's -ffast-math, which I don't know if it get's turned on by -O3 in
gcc. Then there's the SSE2/SSE3 stuff on Intel/AMD hardware, which at
-O3 _may_ be used; MSVC can be comfigured to use those instructions or
not and if you don't work with doubles, you _will_ see changes. (I
seen it happen on Win32 and Win64 here.)

Anyway, when you thought 'long' on 64-bit boxes requires some serious
medication to think about, there's still more wicked little critters
waiting in floating point country. ;-)

Of course, as some books say (e.g.
http://www.network-theory.co.uk/docs/gccintro/gccintro_70.html ) all
these mindnumbing, unpleasant details are 'generally not significant'.
Until they are.


> small changes are not a cause for concern, but the neural net at least
> has a big change- but since the neural network depends on a (random) initial
> condition, it's not that surprising.)

Oh, Neural Net. Well, the random bit in there made it _quite_ hard for
me to get any numbers which could be compared between my various
platforms. Of course, every machine and his sister comes with its own
little runtime library with a 'rand()' function. Great. But not great.
Because the _only_ thing 'standard' with that is that a call will
return a 'semi-random number' to you, period. Somewhat more semi on
some than on others, but that's not the cause for my troubles there.
Basically, everybody uses a different algorithm to produce these
numbers, resulting is _quite_ different NN results when you look at,
for example, Windows machines, compared to Linux boxes. Outside the
Linux/BSD realm, you will very probably hit the same with other
UNIXes. Anyone have an AIX, HPUX or Sun (without gcc) around, just to
name a few?

So what I did after a bit of grumbling, was take out a piece of public
code for one of the Marsaglia random number generators (as those are
fast and have nice distributions) and plug that in. Now at least I
have reproducible random numbers for NN et al (who do not require
cryptographically strong and unpredictable random so the
reproducibility is nice, though one can change that if you like by
adding an explicit seed at the start, say the current time stamp).

What you read here is the same solution to this problem as what is
done inside all professional simulation packages out there. In short:
provide your own random generator, so you get consistent behaviour[*]
across platforms, and 'platforms' includes 'compiler' here. You're
sure to get other results when compiling with, say, Watcom, Borland or
Microsoft, all on the same Windows box. With GCC, different glibc
versions may have different random output, etc.etc.

Care for another worm? ;-)


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

[*] The fact that providing your own random gen also gives you a lot
of _control_ over its behaviour (repetitivity across runs for 100%
reproducible results despite 'random', for example) may be considered
a boon. I do, because a large part of the GerH build work includes
testing against previous and 'vanilla crm114' results, and I want
every difference in there to be 100% explicable.

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php