Re: Mixed 64-bit system GerH binaries / BillYscripts --> two-sided training? YES!

<... sigh> Okay, I'm not really in the mood right now and I need to
get some work /done/ here, but blood goes...

First off:
Bill: good question, but there's another one that should come with it
;-) -- read on. Anyway, your 'initial' answer is:

typedef uint32_t  gimme_32_bits;

or

typedef int32_t gimme_32_bits;

when you want signed (one of the things I wish to play with is a
_signed_ count: + = hammy, - = spammy. Yes, that kills the occurrence
count as it only stores delta, but I want to see what happens, because
from what I see from my files, I might not be interested in occurrence
count for commons at all. Unless someone has a VERY GOOD system to
'kill off' so called 'common words' with high occurrence counts on
BOTH sides of the fence. Nevertheless, the remarks by Fidelis & Paolo
(thank you both) have convinced me I should *rethink* this one because
there's the smell that I am approaching that issue at the wrong level
(count vs. weight&score).
Because score does not rely on a single hash weight (where 'errors'
introduced by the [common word] occurrence counts weighting in on my
score DO ACCUMULATE rapidly!) but on the combined 'hit rates' of
multiple hashes (short message != single word message ;-) ) I believe
the '-1/0/+1' dance for common words weights is negligible, especially
when I apply a 'shaping function' which pushes those low counts down
score-wise into never-neverland.
This idea is all conjecture until tested, so if you think it's
worthless/dumb, you might be bloody right. Just one stupid thing I
want to be able to try, okay? The kid gotta play and burn his fingers
to learn the lesson well.



Additional info for Bill et al:

for regular calculations, use 'int' or 'unsigned int' - when you are
sure the numbers will always fit 32 bit (or less). (Counters, and
stuff.) It's easier on the compiler. There are more CPUs out there
than just Intel x86, who doesn't seem to care that much regarding
'word size' at first glance. It's the exception that, thanks to
Microsoft, occupies all your desks. Yet it is an exception.
Use [u]int32_t only when bit-precise size really matters: that is when
storing or fetching data from (mmap()ed) files, so uint32_t is perfect
for that struct you asked about, Bill, but there's calculus as well
and there the above is applicable. Besides, it'd save you quite a few
typecasts or %" Puint32 " printf() string format hacks when you use
'int', 'long' and 'double' in there. ;-)

And for all who believe 'int is 32 bit': I am very tired now, so
forgive me if you can after this, but suffice to say: some day in the
far away past, Bill was convinced by some folks that it's twin called
'long' would be cross-platform bitsize fixed. Now that we do have
several platforms out there (Windows, UNIXes of all ages, etc.), see
where this got us all. Cause? Did Bill screw up? NYET. Bill was
misinformed. Do me a favor, don't let it happen again.
'int' and 'long' are 'native integer sizes'. Your hardware and
compiler vendors will determine what 'native' _is_ and ISO/ANSI gives
them free reign. Look at it, live with it and adapt.


Second, Bill, I like to suggest you do 'something special' for the
hashes a la GerH builds: I use a separate type for them:

typedef uint32_t crmhash_t;

and everywhere I go and move or calculate hashes, I use that type
'crmhash_t': because hash calculations care a LOT about bit-boundaries
and 'arithmetic overflow': by using that particular type you (a) can
quickly see where the hash matters **as a hash** and (b) switch over
FAST when you move to 64-bit hashes: they're just ONE TYPEDEF away!
:-)) (I tried them already, GerH builds offer different hash algos too
(-mN). It works. ;-) )


When is a 'hash not a hash anymore'?

Example: Calculate a hashtable jump index? hash is not a hash anymore
so code like this:

int index = (hash % table_size);  /* index into table is hash MODULO
tablesize */

use unsigned int if you feel the need (I don't), but I don't mind the
sign bit when I know up front that my indexes will never get beyond
the range 0..2^(N-1) anyhow[*] as the '%' operation is done in
uint32_t size or bigger, only /after/ that the result is 'downgraded'
to 'int' (where N=32 / 64 / ...== 8*sizeof(int)) - and if /any/ of you
feel as anal retentive about it as I sometimes am, then don't even
/try/ to yak about the lacking unsigned again, but kick me where I
_really_ took the shortcut and remind me that I should have coded it
like this instead to be standard 100% compliant AND 100% 32/64/N-bit
system covering, fully irrespective of compiler 'int' size:

size_t index = (hash % table_size);

(The fact nobody mentioned 'size_t' at that time, despite the hefty
discussion here regarding hash table sizes, etc. and signed vs.
unsigned int in the past told me something I did not like to hear.)
And no, gentlemen, 'size_t' is NOT 'unsigned int': your header files
may say it is, but there's also enough of them out there that say it's
an 'unsigned long' which on that particular platform is NOT the same
size as an 'unsigned int'.

[*] for lowest CRM114 supported N=32, worst-case hashtable sizes
beyond 2^31 _elements_ (that's about 2E9 * (4+4) = 16GB(!) filesize
for the /least/ storage-demanding classifiers) are a size that cannot
be memory-mapped on ANY N=32-bit system anyhow - even when your
hardware might support 'bankswitching' or 'segmented addressing' (like
the x86 family), your mmap() equivalent won't ever surpass that 32-bit
boundary on your 32-bit operating system(!) - because any
Harvard-architecture ~ or segment-register augmented 32-bit CPU may
allow access to more than 4GB (= 2^32: 32-bit) bytes by having dual
(code and data) address spaces or a (e.g.) 36-bit address bus
(Pentium, anyone?) for a 32-bit 'word size', the fact of the matter
stays that 'one contiguous addressable block of memory' (like you need
for a hash table/array and get through mmap() et al) will only ever
span the size of your *index* registers - which are stuck at 32 bit
for your 32-bit CPU. That's the difference between RAM on a server
board (which can be as big as 8 or even 16 GB, no sweat) and the
*code* running in that collection of 32-bit CPUs on that same board.
How you code it up to use the segment registers as well to extend your
memory space is a subject not for this ML as it is EXTREMELY
non-portable and IMHO ludicrous to consider for applications like
CRM114.

Before I forget: you wish to cover the 64-bit systems where int is
made 32 bit by the compiler? Excellent. do NOT use 'unsigned int'
because you'll be as wrong as I am in my 'int' laziness. Use 'size_t'
and stick with that when it comes to indexing large [hash]tables. Only
then will you be, err, 'pixel-perfect'.



And last: I would like it when you kept support for C89 compilers in
mind, because MSVC2003/2005 - in use at a LOT of plants - are
definitely NOT C99 to the letter. So are several other compilers. And
besides, sticking to C99 isn't going to save your bacon here either,
it's just a little easier when you want guaranteed bitsize integers
like uint32_t.


I'm going to shut up, you verify my facts.
CRM114 does not benefit from 'Just Works' or 'Easy Answers'. It
benefits from the Complete, Correct Answers, so Bill can evaluate and
pick his vice while he is informed of the consequences. Bill got told
'long' was the Easy Answer always, like abstinence. CRM114 on 64-bit
and Palin's daughter at 17. The real world is not about 'easy, single
answers', folks. And in situations of 'good' (=proper) engineering at
least:
Vertrauen ist gut, Kontrolle ist besser.
Because I mess up too.


Take care and make me proud: take it easier than me, ;-)

Ger







>
>



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/