Re: crm114 on 64 bit SuSE

Thanks for testing and reporting back; much appreciated!

Further responses mingled with original email quotes:

On Sun, Aug 10, 2008 at 2:07 PM, Thomas Spahni <[email protected]> wrote:
[..]
> mail) but the build works well in production. There is just one glitch I
> observed: learning just one message advances the count of messages learned
> by TWO. I still could not find the reason for this. Any pointers would be
> welcome.

Which classifier did you use? (Each keeps track of learn counts
individually, hence the question)


> I tried to install Ger's crm114-20080502-BlameJason-Ger-2423.src.tar as
> well. Compile went well but learning a message with mailreaver results in:

Whoops. Hrmmm... yeah; I didn't test mail*.crm scripts for a long time
as I don't use them myself (I use crm114 for different purposes).
I see where it could produce this kind of barfball, but I am very
short on time now, so fixing this will have to wait unfortunately.
:-((

Mea maxima culpa.


[...]
> Additionally Ger's version produces a significant list of differences in
> megatest (output attached).

Part of the diffs are the 'extra titles' (which show up twice instead
of once) so that I can quickly see which classifiers produced what
diff bits; important when you are looking at slews of numbers.

Let's have a look at your megatest output.
(By the way, did you try 'make check' as well? That is ONLY available
with the GerH builds and runs a growing series of tests which span
more than megatest does. I'm interested to hear what that would
produce on your system...)


> Thank you all for your excellent work and I hope this report may be
> useful.

Absolutely: very useful! Thanks again!


[diffs for vanilla/Bill's CRM114:]
> <  This is CRM114, version 20070807-BlameTheInterns (TRE 0.7.5 (LGPL))
> ---
>>  This is CRM114, version 20070810-BlameTheSegfault (TRE 0.7.5 (LGPL))

A GerH release matching the very latest of Bill's is still pending.
Release will be announced on the list.
This however looks like the megatest_knowngood reference is not up to
date in your vanilla setup as both copyright lines should match
(unless Bill didn't produce a fresh 'knowngood' for the build you
grabbed from the CRM114 site); after all, the 'knowngood' reference is
produced by Bill by running a fresh compiled crm114 binary with the
megatest shell script.

> 206,207c206,207

[...okay...]

> 834,835c834,835
> < CLASSIFY fails; success probability: 0.0307  pR: -14.9943
> < Best match to file #1 (q_test.css) prob: 0.9693  pR: 14.9943
> ---
>> CLASSIFY fails; success probability: 0.0413  pR: -13.6550
>> Best match to file #1 (q_test.css) prob: 0.9587  pR: 13.6550

[differences in floating point values -- when small -- are okay; I use
more double-precision calculations than vanilla code (and less cast
to/from float in places) which has impact on the numbers as well.

Notice those 'off-by-one' diffs in the 'hits' below: these boogers are
still on my list of 'bug indicators' (either in mine, Bill's or both
code repos) and when time allows I want to scratch my head over these
once more. Here the diffs are probably due to bugfixes in the recent
vanilla code, while you seem to check against a somewhat older
megatest_knowngood reference, hence those little buggers showing up.
Vanilla is definitely improving in taste. :-))

The 'rule of thumb' I apply to the diff reports is: integer values
must match _exactly_ unless I have a pretty good excuse; floating
point values may differ a few percent at most -- again, unless I'm
good for it excuses-wise.
Here the diff in pR values beyond that 'few percent' margin is very
probably due to bugfixes in the lastest vanilla/BillY codebase (and
they show up as you seem to be using an older knowngood).


> 837,838c837,838
> < #0 (i_test.css): features: 29641, hits: 6, radiance: 1.21e-03, prob: 3.07e-02, pR: -14.99
> < #1 (q_test.css): features: 26746, hits: 32, radiance: 3.83e-02, prob: 9.69e-01, pR:  14.99
> ---
>> #0 (i_test.css): features: 29641, hits: 7, radiance: 1.65e-03, prob: 4.13e-02, pR: -13.66
>> #1 (q_test.css): features: 26746, hits: 32, radiance: 3.83e-02, prob: 9.59e-01, pR:  13.66

[... and more of the same ...]

>
>
> ----------------------------

And now for the GerH diffs. Good golly...

>
> make megatest (Ger H crm114-20080502-BlameJason-Ger-2423)
[...]
> ==============================================================
> =================== MEGATEST DIFF REPORT =====================
> ===================      (unified)       =====================
> ==============================================================

[... copyright details diffs are a-okay. I just report a bit more to
stdout, which is caught by diff. ...]

[... same applies for (expected) error reports: GerH builds have a
slightly more verbose error reporting facility, hence the diffs. ...]

> - OK_IF_PID_CHANGES: one... MINION PROC PID: 5834 from-pipe: 6 to-pipe: 5
> - OK_IF_PID_SAME_AS_ABOVE: again... MINION PROC PID: 5834 from-pipe: 6 to-pipe: 5
> + OK_IF_PID_CHANGES: one... MINION PROC PID: 2021 from-pipe: 5 to-pipe: 4
> + OK_IF_PID_SAME_AS_ABOVE: again... MINION PROC PID: 2021 from-pipe: 5 to-pipe: 4

[... okay ...]

> @@ -679,29 +695,37 @@
>  one two three
>
>  ***** checking return and exit codes
> +
> +***** checking return and exit codes

[Above is such a 'repeated' title: context diff will onbly show the
duped one, so I get to see what & where quickly. Okay as it is.]

> + ****  Default (SBPH Markovian) classifier

[differences in floating point values -- when small -- are okay; I use
more double-precision calculations than vanilla code (and less cast
to/from float in places) which has impact on the numbers.

Generally spreaking, you should start the headscratching when the
integer numbers don't match exactly. There are a few spots where I
know this will happen (anything which uses the VT (Vector Tokenizer),
but it's always a moment to ponder the state of the union when you see
that (different integer values) happening. Here, there's nothing like
that yet, however, so all is hunky dory.]

>  **** OSB Markovian classifier

A-okay.

> +**** OSB Markov Unique classifier

A-okay

> +**** OSB Markov Chisquared Unique classifier

A-okay

> +**** OSBF Local Confidence (Fidelis) classifier

A-okay

> +**** OSB Winnow classifier

A-okay

> +**** Now verify that winnow learns affect only the named file (i_test.css)

A-okay

> +**** Unigram Bayesian classifier

A-okay

> +**** unigram Winnow classifier

A-okay

> +**** OSB Hyperspace classifier

A-okay

> +**** OSB three-letter Hyperspace classifier

A-okay

> +**** Unigram Hyperspace classifier

A-okay

> +**** String Hyperspace classifier

A-okay

> +**** String Unigram Hyperspace classifier

A-okay

> +**** Vector 3-word-bag Hyperspace classifier
> +
> +crm114: *WARNING*
> +We'd expected to see another value specified for the VT coefficient matrix value but alas, the script apparently didn't deliver enough for that. Next time, make sure you've got a full custom vector!

Ah, yeah. GerH has a slightly different VT codebase: compare the
'megatest_ng.sh' and 'megatest.sh' shell scripts provided with GerH
builds: my VT code requires an extra value for custom vector: specs so
I can pass in dual matrices as well (which is a feature not available
with vanilla VT, so it limits custom vectors for OSB/Markovian
classifiers to a single, shared matrix only. A rather theoretical
point today as OSB/Markov doesn't use VT... yet. Anyway... the 'way
out' is to run 'megatest_ng.sh' to 'fix' this, i.e. use a matching
script which has been adapted to match particular GerH features. (make
megatest_ng)

> -#0 (i_test.css): features: 8809, hits: 0, radiance: 0.00e+00, prob: 1.70e-22, pR: -217.69
> -#1 (q_test.css): features: 8349, hits: 7, radiance: 5.87e-03, prob: 1.00e+00, pR: 217.69

Due to the error above, all the hyperspace tests are nuked.

> +**** Bit-Entropy classifier
[...]
> -#0 (i_test.css): features: 425488 (5%), entropy: 384.969632, jumps: 23, prob: 2.52e-07, pR:  -6.60
> -#1 (q_test.css): features: 552504 (5%), entropy: 225.556532, jumps: 7, prob: 1.00e+00, pR:   6.60
> +#0 (i_test.css): features: 425488 (5%), entropy: 365.245113, jumps: 21, prob: 4.18e-06, pR:  -5.38
> +#1 (q_test.css): features: 552504 (5%), entropy: 235.307962, jumps: 8, prob: 1.00e-00, pR:   5.38

Notice the difference in 'jump' numbers. Definitely a bug. In mine,
Bill's or both? I don't know. But definitely a bug _somewhere_!

> +**** Bit-Entropy Toroid classifier
[...]
> -#0 (i_test.css): features: 425488 (100%), entropy: 459.942001, jumps: 26, prob: 2.95e-07, pR:  -6.53
> -#1 (q_test.css): features: 552504 (100%), entropy: 302.166313, jumps: 10, prob: 1.00e+00, pR:   6.53
> +#0 (i_test.css): features: 425488 (100%), entropy: 519.417393, jumps: 33, prob: 2.39e-09, pR:  -8.62
> +#1 (q_test.css): features: 552504 (100%), entropy: 311.132100, jumps: 11, prob: 1.00e-00, pR:   8.62

Bug mentioned above permeates all of the Entropy classifier (which
makes kinda sense).

> +**** Fast Substring Compression Match Classifier

A-okay

> +**** Neural Network Classifier
[...]
> -CLASSIFY fails; success probability: 0.241661  pR: -62.5048
> -Best match to file #1 (q_test.css) prob: 0.7583  pR: 62.5048
> +CLASSIFY fails; success probability: 0.338256  pR: -6.5569
> +Best match to file #1 (q_test.css) prob: 0.6617  pR: 6.5569

As you can see, the pR number differences are _significant_. NN
classifier is still tagged as 'experimental' I believe, and has seen
some significant change the last few months. You may consider
'broken'; at least it's not stable enough for production purposes.

> +**** Alternate Neural Network Classifier test script
[...]
> -CLASSIFY fails; success probability: 0.017262  pR: -282.7380
> -Best match to file #1 (q_test.css) prob: 0.9827  pR: 282.7380
> +CLASSIFY fails; success probability: 0.018972  pR: -281.0282
> +Best match to file #1 (q_test.css) prob: 0.9810  pR: 281.0282
>  Total features in input file: 195
> -#0 (i_test.css): prob: 1.73e-02, pR: -282.74
> -#1 (q_test.css): prob: 9.83e-01, pR: 282.74
> +#0 (i_test.css): prob: 1.90e-02, pR: -281.03
> +#1 (q_test.css): prob: 9.81e-01, pR: 281.03

This looks much better, like it should, but I still don't feel too
confident. Nice to see the numbers being close together, but I
wouldn't bank on this. Yet.

> +**** Support Vector Machine (SVM) unigram classifier

Wow! Matching numbers (integers). Haven't seen those in a while for
this one. Looks okay...

> +**** Support Vector Machine (SVM) classifier

And another 'wow!'

> +**** String Kernel SVM (SKS) classifier
[...]
> +**** String Kernel SVM (SKS) Unique classifier

And add to that a double 'simply amazing!' Of course, it might just be
me and my geriatric brain that only recalls slews of diffs here, but
this _really_ looks _very_ nice!
All integer values match up to the last bit and very slight diffs in
the floating point values. Hadn't expected to see that for these
fellas...


> +**** Bytewise Correlation classifier
[...]
> -#0 (INTRO.txt): features: 53186, L1: 219063 L2: 325594 L3: 585264, l4: 1747198 prob: 9.34e-03, pR:  -2.03
> -#1 (QUICKREF.txt): features: 69063, L1: 385852 L2: 599956 L3: 6579582, l4: 326320000 prob: 9.91e-01, pR:   2.03
> +#0 (INTRO_mt_ng_reference_2.input): features: 53186, L1: 219063 L2: 325594 L3: 585264, L4: 1747198 prob: 9.34e-03, pR:  -2.03
> +#1 (QUICKREF_mt_ng_reference_1.input): features: 69063, L1: 385852 L2: 599956 L3: 6579582, L4: 326320000 prob: 9.91e-01, pR:   2.03

Ah, yes, the dreaded feature count discrepancies. Bugs one way or
another. Consider classifier broken until further notice.

>  **** Clump / Pmulc Test

And the MAJOR crufticraft in GerH. Well,
> +crm114: *UNTRAPPABLE ERROR*
says it all, doesn't it?

The 'fun' thing is that I never had a successful CLUMP/PMULC run on
any of my machines (AMD @ 32-bit Win/64-bit Win+Linux) while you seem
to get a very nice output for it on an Intel box @ 64-bit, while a
(old) code review says you shouldn't. :-S
Of course, it might just be due to the latest code in your
vanilla/Bill crm114 code copy, but one thing's for sure: GerH is
broken in regard to this one, for sure.

> ==============================================================
> =================== MEGATEST DIFF REPORT =====================
> ===================      (regular)       ====================
> ==============================================================

As above (just another diff format for the same test run): notice that
those 'titles' pop up in here as well, so you can quickly see which
test section produced what bit of diff.

[...]

Thank you for the testing and reporting. It makes me happy to see that
others are checking this material thoroughly.


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/