Re: Mixed 64-bit system GerH binaries / BillYscripts --> two-sided training? YES!

On Wed, Sep 3, 2008 at 7:45 PM, Bill Yerazunis <[email protected]> wrote:
>   BOTH sides of the fence. Nevertheless, the remarks by Fidelis & Paolo
>   (thank you both) have convinced me I should *rethink* this one because
>   there's the smell that I am approaching that issue at the wrong level
>   (count vs. weight&score).
>
> Indeed.  Absolute count _does_ matter and there is information
> hidden even in something with very high and almost equally
> distributed counts.  This was demonstrated most recently by
> OSBF winning the CEAS filter contest.
>
> OSBF does a very good job of estimating the actual significance
> of a relatively small number of instances.  I think the actual
> exact answer is to be found by some manipulation of the binomial
> theorem.

A-ha. Thanks for that elaboration. Ok, I'll keep that in mind when I
restart digging into that bit again. Got to try a few different things
then and do some serious peering at frothing test tubes, I expect. :-)


>   typedef uint32_t crmhash_t;
>
> That's a good idea.  Hard-define only what must be hard_defined.

The benefit of a typedef there is that makes the code a bit more
readable (a lot of ints or uint32_t floating around can cloud the eye)
and since you have been considering 64-bit hashes in the past as well,
this is the simplest way to 'prepare' for moving towards those.

Doesn't mean one should go 'overboard' on typedef'ing every int around
though; hashes merit a typedef IMO as they are very pervasive
throughout the CRM114 system and are as such a 'special case'. Just a
note, though you will find in some environments there's whole
religious wars fought over that one alone, especially in the 'company
coding standards' arena. Coding an O(n^3) routine where a simple O(n)
alternative is readily available is a-ok, but dare to put the
semicolon on the wrong line and it's your saturday down the tubes to
adhere. :-))
For that bit, I got a new answer: uncrustify. It's my modern day GNU
indent. Not perfect, but /very/ neat. Unfortunately it doesn't have a
flag for your '};' style yet. ;-) We might have to see how to fix that
when backmerging GerH code chunks into the mainline.

>   int index = (hash % table_size);  /* index into table is hash MODULO
>   tablesize */
>
> OK, if hash is either an uintNN_t or an "unsigned long" I can see how that
> would always work.

Remember: hashes require a very particular integer math *overflow*
behavior (dropping top bits you don't ever want to see appear again
just because you do a right-shift operation, for example). In such
circumstances, the [u]intN_t types are the only good choice as they
guarantee that particular behavior at specific bit positions. (compare
to cryptographic hashes which also need these particular types. For
instance, OpenSSL has a lot of those and it does it's stinkin' best to
find the proper type to use for all those platforms that are not 100%
C99 yet. You're not into crypto but you need the same shift, add, etc.
behavior inside the hash calculus (and consequently its 'extension',
the Vector Tokenizer (as it mixes hashes, which is a hash production
operation in itself again).


> OK, so what is a size_t then?  A type designed to be able
> to hold the size of the largest array of the smallest item that
> this compiler on this machine can generate?

;-) Yep. The definition is that a size_t is the integer type which
will be able to contain all different values which result from pointer
subtraction. So:

type *p1;
type *p2;

...

l = p2 - p1;

implies variable l should have type size_t. To be more specific: p2-p1
produces an integer value and type which matches size_t; if you do not
wish to get warnings about data loss (bit stripping), l should have
the same (or larger) type as produced by the p2-p1 operation. All
compilers since the birth of ANSI C come with the custom 'size_t'
typedef in their runt-time library header files.

A good compiler will warn you about probable loss of
data/precision/bits if, say, l is int (because it is signed, you
ditched one power of 2 in available range at least, assuming size_t is
unsigned int. Of course, if size_t is 'unsigned long' it's worse when
sizeof(long) > sizeof(int) -- standards say only that sizeof(long) >=
sizeof(int), btw).
In GCC you may have to dial up the warnings to a 'pedantic' level
(-Wall / -pedantic) to get to see this all; Microsoft has a special
compiler option which causes the compiler to issue additional '64-bit
issue' related warnings.

Personally, I generally use 'int' or 'unsigned int' because I'm not
always _that_ adherent, but I always check the expected value range.
For instance, strlen(s) will also produce a 'size_t' typed return
value (see your system header files: all the str*() and mem*() that
return a distance/length/offset, produce a size_t return value), but I
often use 'int' (and thus cast when I want to get rid of the warning)
when this applies to filepaths and such (or CRM scripts ;-) ) because
I am willing to risk that nobody is going to feed my code a > 2GB
script on _any_ platform and I like 'int' more than 'size_t', but
that's pure emotion. That means that the GerH code is only _somewhat_
cleaner type-wise, but it's not ANSI/ISO pedigree quality, so you can
look and see what I did, but it's not on par with gospel, okay?



> OK.  Got it.  All hash-calculated indices shall be size_t type.

Indeed, for those, it's [almost] mandatory. GerH code today does not
do this and is thus 'faulty' for those 64-bit operating systems which
come with compilers that offer 32-bit int (and 64-bit long) but
provide a mmap() which can map > 4G element-sized files nevertheless
-->
since the GerH builds use int till now, my code does NOT generally
SUPPORT > 8GB (signed int @ 31 bit * 4 bytes per hash) CSS files on
64-bit boxes: I was lazy there in some spots and I screwed that bit
up. (GCC 4.0 on AMD64/Suse10.2 has int=4, long=8 bytes)
Side Note: 64-bit CPU doesn't automatically mean we've got a 64-bit OS
as well; see lot of Windows boxes: 32-bit Windows on 64-bit hardware -
even with Vista nowadays. 'platform' in my definition is the combo:
(1)CPU, (2)operating system and (3)compiler. Each brings its own
little wickedness to the party.


> Thanks, Ger.  I'll put that into the coding spec.

You're welcome. ;-)




-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/