Re: Can some one enlight me regarding the results of megatest?

On Wed, Sep 3, 2008 at 7:46 PM, Steve <[email protected]> wrote:
> I understand those with OK_IF_mumble but ...

ok, that's one <snip> coming up then:

On Wed, Sep 3, 2008 at 7:46 PM, Steve <[email protected]> wrote:
> Hello list

<snip>

> 980a981,990
>>
>> ./crm114: *WARNING*
>>  neural: failed to converge within the training limit.   Beware your results.
>>  You might want to consider a larger network as well.
>> I'll try to keep working.
>> This happened at line 2 of file (from command line)
>> (runtime system location: crm_neural_net.c(1678) in routine: crm_neural_net_learn)
>> The line was:
>> --> learn < neural refute fromstart > (q_test.css)
>> <--

Well, this baby is no fun. It basically says the Neural Net
(experimental!) classifier didn't pan out on your machine. Conclusion:
everything up to NN in Megatest (that's OSB, OSBF, Markovian,
Hyperspace and a few others) is hunky dory because you didn't get any
diff for those, so you should /not/ use NN in your scripts, but the
others preceeding that are okay.

Because NN train went down the tube here, the subsequent NN [classify]
tests are going to b0rk as well by giving you pR numbers that are way
off:

> 982,983c992,993
> < CLASSIFY fails; success probability: 0.241661  pR: -62.5048
> < Best match to file #1 (q_test.css) prob: 0.7583  pR: 62.5048
> ---
>> CLASSIFY fails; success probability: 0.324688  pR: -7.7780
>> Best match to file #1 (q_test.css) prob: 0.6753  pR: 7.7780

That's: expected -62/+62 but got -7.7/+7.7 from the test instead. My
rule of thumb: more than 5 percent off, so defcon 3 for that
classifier right there.

Unfortunately, the diff doesn't tell you the Classifier this way so
it's kinda hard to see which classifier we're talking about here, but
given the NN b0rk just before, I'm heavily betting on this being NN
classify going pearshaped as a result.


> 985,986c995,996
> < #0 (i_test.css): prob: 2.42e-01, pR: -62.50
> < #1 (q_test.css): prob: 7.58e-01, pR:  62.50
> ---
>> #0 (i_test.css): prob: 3.25e-01, pR:  -7.78
>> #1 (q_test.css): prob: 6.75e-01, pR:   7.78

Same.

> 989,990c999,1000
> < CLASSIFY succeeds; success probability: 0.889200  pR: 189.2001
> < Best match to file #0 (i_test.css) prob: 0.8892  pR: 189.2001
> ---
>> CLASSIFY succeeds; success probability: 0.521953  pR: 0.2195
>> Best match to file #0 (i_test.css) prob: 0.5220  pR: 0.2195
> 992,993c1002,1003
> < #0 (i_test.css): prob: 8.89e-01, pR: 189.20
> < #1 (q_test.css): prob: 1.11e-01, pR: -189.20
> ---
>> #0 (i_test.css): prob: 5.22e-01, pR:   0.22
>> #1 (q_test.css): prob: 4.78e-01, pR:  -0.22

Same.

Check the makefile for that diff statement (the reference file is
'megatest_knowngood.log'): you may want to try 'diff -u
megatest_knowngood.log megatest.log' instead, maybe even with extra
'-C5' or '-B5' / '-B10' commandline option, to see the comment lines
reporting which classifier is tested now as well. It helped me a lot
when I was working on the Windows & 64-bit code and looking for these
differences.


> 999,1000c1009,1010
> < CLASSIFY fails; success probability: 0.017262  pR: -282.7380
> < Best match to file #1 (q_test.css) prob: 0.9827  pR: 282.7380
> ---
>> CLASSIFY fails; success probability: 0.002674  pR: -297.3258
>> Best match to file #1 (q_test.css) prob: 0.9973  pR: 297.3258

This is -282/282 against -297/297. It's a judgment call really (Bill
may say this is okay), but I am more(?) strict as this is too much
(297-282 = ~ 5%) for me to feel 'safe' about it, despite that 'rule of
thumb' I mentioned earlier. Wouldn't be a 'preferred classifier' for
me, because I would know I'd get (slightly) different results for
different platforms.

pR is a derivative of the probability there, so I don't really look at
that number anymore - unless other things spook me. Here, pR is too
far of for my tastes so I skip prob. as it's surely be 'wrong' too.

I'm not sure which classifier this would be (don't know the expected
pR numbers by head ;-) --> diff -u )

> 1002,1003c1012,1013
> < #0 (i_test.css): prob: 1.73e-02, pR: -282.74
> < #1 (q_test.css): prob: 9.83e-01, pR: 282.74
> ---
>> #0 (i_test.css): prob: 2.67e-03, pR: -297.33
>> #1 (q_test.css): prob: 9.97e-01, pR: 297.33

As above. A bit of doubt. (-297 vs. -282)

FYI: you are at least looking at diffs for EXPERIMENTAL classifiers
anyhow (because megatest first tests the 'production classifiers' and
then moves on to testing the experimental ones, so you might consider
the whole bunch 'not really relevant' if you are going for 'production
classifiers only'.

> 1006,1007c1016,1017
> < CLASSIFY succeeds; success probability: 0.971915  pR: 271.9152
[...]
> 1018,1019c1028,1029
> < #0 (i_test.css): documents: 300, features: 7464,  prob: 8.90e-01, pR:   9.09
> < #1 (q_test.css): documents: 175, features: 6223,  prob: 1.10e-01, pR:  -9.09
> ---
>> #0 (i_test.css): documents: 300, features: 7464,  prob: 8.87e-01, pR:   8.96
>> #1 (q_test.css): documents: 175, features: 6223,  prob: 1.13e-01, pR:  -8.96

Okay, this one is nice for some additional hints&tips. It's okay
(clearly within the 5% boundary I took - not a sure thing, but
definitely a 'probably all right'.

What I do /then/ (i.e. when the pR 'looks good') is check all the
'integer numbers': document counts, feature counts, etc. and those
MATCH UP EXACTLY.
If they do NOT (and you will find that GerH builds are off-by-a-few
for several classifiers) it means there's a bug in there, or at the
very least a feature difference. Which is a showstopper until you can
get a pretty good explanation why. (For instance, my document counts
are of-by-1 in a few classifiers, because there's a bug fix in GerH
which isn't all there in the mainline (has to do with <unique> and
feature series for particular classifiers); other count differences
are due to the fact that I rewrote the VT code a long time ago and it
will produce more feature hashes for boundary cases. The latest
mainline is getting much closer to my numbers, though.  Another bit
Bill & I need to go through when time allows. As to which is 'better'
then? That depends on the bugfix being actually a fix or a tweak due
to a misconception. Of course, I am opinionated, but that's not what
this is about. It about this:

if you see any differences in feature counts or documents (integer)
numbers reported, that is enough reason to ask around on the ML if
it's okay. It probably is (there are types of bugs, which can be
tolerated some times; those bugs are about accuracy instead of
operation -- that's also where most of the Blame is shifting around
for. :-) )


> 1022,1023c1032,1033
[...]
>> #1 (q_test.css):documents: 175, features: 37599,  prob: 1.53e-02, pR: -18.10
> ------------------------------------

Bill can probably dream these stats, but I need a diff -u where the
classifier print lines show up as well (-C5 ? maybe even -C10 ? or
better yet: -B10 for the diff command?); you might benefit from that
too.

Anyway, I hope the explanation up there helped a bit in 'reading' the stats.


> And what about the others like this one:
[...]
> The pR seems to be equal but the probability digits are not the same. Is this an issue?

See above: some are an issue, some are not. 5% on pR is a ballpark
figure. Can use that on the 'probability' floating point value as well
(that's always 5% of the largest value of the two, by the way).



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/