Re: Mixed 64-bit system GerH binaries / BillYscripts --> two-sided training? YES!

   From: "Ger Hobbelt" <[email protected]>

   <... sigh> Okay, I'm not really in the mood right now and I need to
   get some work /done/ here, but blood goes...

   First off:
   Bill: good question, but there's another one that should come with it
   ;-) -- read on. Anyway, your 'initial' answer is:

   typedef uint32_t  gimme_32_bits;

   or

   typedef int32_t gimme_32_bits;

   when you want signed (one of the things I wish to play with is a
   _signed_ count: + = hammy, - = spammy. Yes, that kills the occurrence
   count as it only stores delta, but I want to see what happens, because
   from what I see from my files, I might not be interested in occurrence
   count for commons at all. Unless someone has a VERY GOOD system to
   'kill off' so called 'common words' with high occurrence counts on
   BOTH sides of the fence. Nevertheless, the remarks by Fidelis & Paolo
   (thank you both) have convinced me I should *rethink* this one because
   there's the smell that I am approaching that issue at the wrong level
   (count vs. weight&score).

Indeed.  Absolute count _does_ matter and there is information 
hidden even in something with very high and almost equally 
distributed counts.  This was demonstrated most recently by
OSBF winning the CEAS filter contest.

OSBF does a very good job of estimating the actual significance
of a relatively small number of instances.  I think the actual
exact answer is to be found by some manipulation of the binomial
theorem.

   Additional info for Bill et al:

   for regular calculations, use 'int' or 'unsigned int' - when you are
   sure the numbers will always fit 32 bit (or less). (Counters, and
   stuff.) It's easier on the compiler. There are more CPUs out there
   than just Intel x86, who doesn't seem to care that much regarding
   'word size' at first glance. It's the exception that, thanks to
   Microsoft, occupies all your desks. Yet it is an exception.
   Use [u]int32_t only when bit-precise size really matters: that is when
   storing or fetching data from (mmap()ed) files, so uint32_t is perfect
   for that struct you asked about, Bill, but there's calculus as well
   and there the above is applicable. Besides, it'd save you quite a few
   typecasts or %" Puint32 " printf() string format hacks when you use
   'int', 'long' and 'double' in there. ;-)

   And for all who believe 'int is 32 bit': I am very tired now, so
   forgive me if you can after this, but suffice to say: some day in the
   far away past, Bill was convinced by some folks that it's twin called
   'long' would be cross-platform bitsize fixed. Now that we do have
   several platforms out there (Windows, UNIXes of all ages, etc.), see
   where this got us all. Cause? Did Bill screw up? NYET. Bill was
   misinformed. Do me a favor, don't let it happen again.
   'int' and 'long' are 'native integer sizes'. Your hardware and
   compiler vendors will determine what 'native' _is_ and ISO/ANSI gives
   them free reign. Look at it, live with it and adapt.

   Second, Bill, I like to suggest you do 'something special' for the
   hashes a la GerH builds: I use a separate type for them:

   typedef uint32_t crmhash_t;

That's a good idea.  Hard-define only what must be hard_defined.

   and everywhere I go and move or calculate hashes, I use that type
   'crmhash_t': because hash calculations care a LOT about bit-boundaries
   and 'arithmetic overflow': by using that particular type you (a) can
   quickly see where the hash matters **as a hash** and (b) switch over
   FAST when you move to 64-bit hashes: they're just ONE TYPEDEF away!
   :-)) (I tried them already, GerH builds offer different hash algos too
   (-mN). It works. ;-) )


   When is a 'hash not a hash anymore'?

   Example: Calculate a hashtable jump index? hash is not a hash anymore
   so code like this:

   int index = (hash % table_size);  /* index into table is hash MODULO
   tablesize */

OK, if hash is either an uintNN_t or an "unsigned long" I can see how that
would always work.

   use unsigned int if you feel the need (I don't), but I don't mind the
   sign bit when I know up front that my indexes will never get beyond
   the range 0..2^(N-1) anyhow[*] as the '%' operation is done in
   uint32_t size or bigger, only /after/ that the result is 'downgraded'
   to 'int' (where N=32 / 64 / ...== 8*sizeof(int)) - and if /any/ of you
   feel as anal retentive about it as I sometimes am, then don't even
   /try/ to yak about the lacking unsigned again, but kick me where I
   _really_ took the shortcut and remind me that I should have coded it
   like this instead to be standard 100% compliant AND 100% 32/64/N-bit
   system covering, fully irrespective of compiler 'int' size:

   size_t index = (hash % table_size);

   (The fact nobody mentioned 'size_t' at that time, despite the hefty
   discussion here regarding hash table sizes, etc. and signed vs.
   unsigned int in the past told me something I did not like to hear.)
   And no, gentlemen, 'size_t' is NOT 'unsigned int': your header files
   may say it is, but there's also enough of them out there that say it's
   an 'unsigned long' which on that particular platform is NOT the same
   size as an 'unsigned int'.

OK, so what is a size_t then?  A type designed to be able
to hold the size of the largest array of the smallest item that
this compiler on this machine can generate?

   [*] for lowest CRM114 supported N=32, worst-case hashtable sizes
   beyond 2^31 _elements_ (that's about 2E9 * (4+4) = 16GB(!) filesize
   for the /least/ storage-demanding classifiers) are a size that cannot
   be memory-mapped on ANY N=32-bit system anyhow - even when your
   hardware might support 'bankswitching' or 'segmented addressing' (like
   the x86 family), your mmap() equivalent won't ever surpass that 32-bit
   boundary on your 32-bit operating system(!) - because any
   Harvard-architecture ~ or segment-register augmented 32-bit CPU may
   allow access to more than 4GB (= 2^32: 32-bit) bytes by having dual
   (code and data) address spaces or a (e.g.) 36-bit address bus
   (Pentium, anyone?) for a 32-bit 'word size', the fact of the matter
   stays that 'one contiguous addressable block of memory' (like you need
   for a hash table/array and get through mmap() et al) will only ever
   span the size of your *index* registers - which are stuck at 32 bit
   for your 32-bit CPU. That's the difference between RAM on a server
   board (which can be as big as 8 or even 16 GB, no sweat) and the
   *code* running in that collection of 32-bit CPUs on that same board.
   How you code it up to use the segment registers as well to extend your
   memory space is a subject not for this ML as it is EXTREMELY
   non-portable and IMHO ludicrous to consider for applications like
   CRM114.

   Before I forget: you wish to cover the 64-bit systems where int is
   made 32 bit by the compiler? Excellent. do NOT use 'unsigned int'
   because you'll be as wrong as I am in my 'int' laziness. Use 'size_t'
   and stick with that when it comes to indexing large [hash]tables. Only
   then will you be, err, 'pixel-perfect'.

OK.  Got it.  All hash-calculated indices shall be size_t type.


   And last: I would like it when you kept support for C89 compilers in
   mind, because MSVC2003/2005 - in use at a LOT of plants - are
   definitely NOT C99 to the letter. So are several other compilers. And
   besides, sticking to C99 isn't going to save your bacon here either,
   it's just a little easier when you want guaranteed bitsize integers
   like uint32_t.


   I'm going to shut up, you verify my facts.
   CRM114 does not benefit from 'Just Works' or 'Easy Answers'. It
   benefits from the Complete, Correct Answers, so Bill can evaluate and
   pick his vice while he is informed of the consequences. Bill got told
   'long' was the Easy Answer always, like abstinence. CRM114 on 64-bit
   and Palin's daughter at 17. The real world is not about 'easy, single
   answers', folks. And in situations of 'good' (=proper) engineering at
   least:
   Vertrauen ist gut, Kontrolle ist besser.
   Because I mess up too.

   Take care and make me proud: take it easier than me, ;-)

Thanks, Ger.  I'll put that into the coding spec.

	- Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/