Re: GerH source distro at hebbut.net: HOW TO compile/compare/merge for Bill et al --> + megatest.sh timings on AMD64

On Sun, Mar 30, 2008 at 9:49 PM, Paolo <[email protected]> wrote:
>  hm, I dunno ... tried shared libs only and it's fine :
>
>  # ldd /usr/src/crm114-20080326-BlameSentansoken/src/crm114
>         libtre.so.4 => /usr/lib/libtre.so.4 (0xb7faf000)
>         libm.so.6 => /lib/tls/libm.so.6 (0x41150000)
>         libreadline.so.5 => /lib/libreadline.so.5 (0xb7f81000)
>         libc.so.6 => /lib/tls/libc.so.6 (0x41019000)
>         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x41000000)
>         libncurses.so.5 => /lib/libncurses.so.5 (0xb7f42000)

It should, that's what in use on my own box(es). The 'static' build
never worked for me (and with that I mean also the builds from
straight vanilla BillY). Something to recheck,

>  > the .inpout files _must_ be there - the fact that they're not means I
>  > b0rked the releases. Why the .input's? Because _any_ change to those
>
>  yup, there's something screwd - there are several megatest*.sh, in src/
>  and tests/, many *.input but in tests/, but not those QUICKREF/INTRO:

Found the issue; missing in the Makefiles :-(((((

FYI: to keep it 'compatible' with BillY vanilla, the 'master' of
everything is in ./src/ so when Bill hopefully decides to take a stab
at this, he can 'cd src' and do his regular thing (though this time
with '-f Makefile.BillY' ;-) ).

The tests/ and other dirs are periodically updated (manually) by
running the ./distribute_yada_yada_yada.sh shell script available in
the distro - long name I just forgot, but you'll recognize it when you
see it.


>  tests/COLOPHON_mt_ng_reference_3.input
>  tests/FAQ_mt_ng_reference_4.input
>  tests/Makefile
>  tests/Makefile.am
>  tests/Makefile.in
>  ...

Those made it into ./tests/ while there should be some more.

I've been lazing, because all I test regularly is the 'BillY
semi-compatibility', by running

cd src
make test

or other test scenarios in ./src/ itself, while I _should_ have been
running them in ./tests/ too.

As said before, 'make check' has only got a _bit_ of attention since
build 1960, before that, it was just trying to tag along but failing
dramatically.
Even with build 1960, it's still broken.


>  > text files (and I added and fixed a bits in there) will result in
>  > megatest to report (quite) different numbers. You don't want that when
>
>  yes, but not that much - but of course we don't want to compare with
>  different inputs.

I agree, but I find that little differences are sometimes VERY
important. I had only little differences in Bit Entropy output and it
turned out there was some 'little' details totally screwed up when
running on a 64-bit platform.
Anyway, to 'prove' a GerH build is _identical_ to vanilla BillY, I
need it to be able to produce the _exact_ same results as a
megatest_knowngood, even when those results are due to flaws. In its
current form, you'll need to tweak a few #defines to make it so
(Hyperspace for one), but such _identical_ output at least takes away
a _huge_ chunk of reasonable doubt when fixes/changes are presented.
Change one typo in those txt files and you'll _never_ get the results
you had before: 'hard numbers' like 'feature count', etc. - i.e. the
non-floating point counters in crm114 - _will_ be off when you change
the learned inputs, even by one character. I know, because I've been
there. Several times over.

So what I did with the very latest builds is allow me to update those
text files as the documentation they are, while giving the tests their
own strictly defined inputs, so that I can compare outputs properly.
One of the changes there, is that CR vs CRLF line termination is also
very important, especially when you move from a LF-based UNIX
environment to a Win32 CRLF-based env: those tests will _never_ match
up unless you treat your inputs as BINARY _files_. (The results are
still off because it looks like redirected input from file still does
some crlf comversion mumbojumbo on my inputs :-((   now all I need to
find is _where_ the damn thing screws up.)



>  > The ULTRA long run times make me wonder. I've found similar results
>
>  local hdd, it's all bounded to CPU/algo problems, if any: but re-checking
>  now those numbers: (Piii time) / (XP2800 time) ~ 400/17 ~ 24 - so in this
>  test an Athlon XP2800 clocked ~2GHz would be ~24 times faster than a Piii
>  450MHz: I think it's a reasonable figure, so the ~400 mins shouldn't raise
>  a flag here - as long as the ~17 mins run time on the other CPU sounds ok,
>  that is.

Ran 'time make test' to be sure, because I _thought_ it was somewhere
around 5', but on a (to be precise) 3.2 GHz AMD64, SuSe 10.2,
completely UNloaded machine, it took the darn thing real 11m40, user
10m5. Everything local; dedicated fast harddisk for the CSS files and
tests, and all. Turns out NN is even slower than I thought it was.

Close enough?

Overview of timing test series:

real/user times:

a) megatest: the whole thing and nothing but the thing:

11m40 / 10m5

b) megatest: anything 'neural' (NN) kicked out:

0m58 / 0m36

woohoo!

c) megatest: all experimentals kicked, EXCEPT (because I like 'em)
Hyperspace and BitEntropy:

0m49 / 0m27

d) megatest: all experimentals gone, i.e. anything following WINNOW is ditched:

0m39 / 0m26


So either I take a profiler to NN and see if I can come up with
something a little faster (not very likely unless I've a Moment), OR
tell the unsuspecting 'make test' user it's time for a coffee and
there's time for the morning paper to spare.



>  yep, better document/update roughly the expected times - waiting ~7 h for
>  something expected to complete in ~1' raise some doubt ;)
>  And that should take into account per-classifier/cumulative wrt included
>  classifiers - that "1'" was stated when only (now) 'production' classifiers
>  were supposed to run.

See above. I'm of two minds there.


>  > it should do this:
>  > cd src
>  > make megatest_ng
>
>  nope, it does:
>   (cd src && $(MAKE) $(AM_MAKEFLAGS) megatest);
>  then
>   $(srcdir)/megatest.sh 2 > megatest_test.log 2>&1

urgh? The Makefile magic in there is hunky dory, but at _least_ it
should call megatest_ng.sh (note the '_ng'). The '2' argument is a
little addition of mine: it makes the script print those 'output
/.../' statements TWICE, so I get to see them in a diff. Twas rather
difficult to spot _which_ classifier was barfing up _which_ slightly
different hairball, when onlt the diffs in pR are shown, so I made
sure I can now see which classifier test section megatest.* has been
working on at the time.


>  > because megatest_ng is the one using those input files (some really
>  > wicked little differences in input feed when redirecting on UNIX and
>
>  which seem supposed to be in src/
>
>  #--[src/megatest_ng.sh, src/megatest.sh]---
>  ...
>   ./crm114 '-{learn ...:]]+/}' < QUICKREF_mt_ng_reference_1.input
>  ...
>
>  but I think they should be sourced from ../tests/.

See further up: ./src/ is 'master' for ./tests/ so I can stick to
BillY dev environment compatibility. Fat chance, but one can hope. ;-)




-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace