Re: Behavior divergence in makefile vs. "in person".
"Ger Hobbelt" <[email protected]>
| Newsgroups | gmane.mail.spam.crm114 |
|---|---|
| Message-ID | <[email protected]> |
Welcome to the 64-bit club. ;-)
Haven't checked lately, but NN has most often been this flaky for me
(my current code here 'works' (as in: does not crash), but that's no
help as I replaced the random generator to ensure cross-system
reproducible results with CRM114 -- after all, GerH builds have
automated reference comparison checks for each of the tests, and NN is
very sensitive to different random gens: each system has it's own.
I recall the many times NN barfed out on me fatally, as now happens to
you. Whenever this happened, both commandline and make runs produced
the same fatal result.
One thing I can imagine is a changed stack limit or other 'tighter'
memory protection settings inside make. Note though that I strongly
suspect your /commandline/ from not showing what is actually wrong;
for instance, it's already for a VERY long time that
-{learn < neural refute fromstart > (q_test.css) /[[:graph:]]+/} ...
blows up over here. And when it did, it was often system dependent
here (Win32 / Win64 / SuSE64). This was mentioned before when I was
working on crm114 intensively half a year or more ago.
I just now quickly checked my own tests/Makefile.am and grep for
CRM114_CHECK_OVERRIDE=0 in there hints:
that's equivalent to the third NN line in your megatest.sh coredumping
and continue on down from there, as I experienced it. Of course, those
two learns blowing a gasket leads to disaster for subsequent classify,
so the next chance at a non-crashing trial is test #2:
# **** Alternate Neural Network Classifier test script
**********************************************************
Neural_Network_test2: $(TEST_PREREQUISITES)
test_timing_minutes_announcement alternating_example_neural.crm
$(E) "*******************************************"
$(E) "* Alternate Neural Network Classifier test script "
$(E) "*******************************************"
-$(SILENT)rm -f i_test.css
-$(SILENT)rm -f q_test.css
$(SILENT)\
CRM114_CHECK_FILTER_ARGS="-v prob=0.2 -v pR=10.0"; export
CRM114_CHECK_FILTER_ARGS; \
./testscript.sh alternating_example_neural.crm "" -
"$(builddir)/crm114_classify.filter.sh"
which tells us that, for GerH crm114, this one is within +/- 10.0
(probably about 30% of that range: +/-3) range of your megatest.sh
reference results. And does NOT crash (or it'd have had another
CRM114_CHECK_OVERRIDE=0 as well in there. The other ('older') NN test
fails dramatically often enough, like you mention now.
I checked, and currently test #1 doesn't dump core like yours, but the
latest WGET for NN I see is from around summer this year, so I am not
sure we're anywhere near to 'in sync'. Relatively speaking.
On the one hand I feel your pain in tracking this bastard down, on the
other hand I am somehow glad you hit these: just maybe, finally these
pieces of code will get the attention they deserve.
Anyway, let's see if we can get you anywhere near a 'working' system
again. Here's the main (important) differences between GerH and your
wget:
1) I replaced the random generator - and NN is VERY sensitive to that
one. Each system has its own and that's making matters very bad for
automated testing setups like mine: GerH 'make tests' actually
verifies the tests and those differences in rand() on the various
boxes leads to WILDLY varying NN output. Currently mine says:
-------------snippet-----------
TEST SCRIPT + COMMANDLINE:
-{learn < neural refute fromstart > (i_test.css) /[[:graph:]]+/}
RETURN CODE:
0
STDERR TEST OUTPUT:
crm114: *WARNING*
neural: failed to converge within the training limit. Beware your results.
You might want to consider a larger network as well.
I'll try to keep working.
This happened at line 2 of file (from command line):
learn < neural refute fromstart > (i_test.css) /[[:graph:]]+/
(runtime system location: crm_neural_net.c(2208) in routine:
crm_neural_net_learn)
STDOUT TEST OUTPUT:
TEST SCRIPT + COMMANDLINE:
-{ isolate (:s:); {classify < neural > ( i_test.css | q_test.css )
(:s:)/[[:graph:]]+/ ; output / type I :*:_nl::*:s::*:_nl:/} alius {
output / ty
pe Q :*:_nl::*:s::*:_nl:/ } }
RETURN CODE:
0
STDERR TEST OUTPUT:
STDOUT TEST OUTPUT:
type Q
CLASSIFY fails; success probability: 0.227734 pR: -76.3110
Best match to file #1 (q_test.css) prob: 0.7723 pR: 76.3110
Total features in input file: 268
#0 (i_test.css): icnr: 0.17 ocnr: 0.64 prob: 2.28e-01, pR: -38.34
#1 (q_test.css): icnr: 0.88 ocnr: 0.08 prob: 7.72e-01, pR: 76.31
-----------------------------------
Jack in another seed or another high quality RNG (or a low quality
one, whatever) and that warning may go away, the coredump can rear its
head again for the refute-learn lines (and onwards of course), etc.
etc. Options galore.
2) a quick diff between wget and GerH shows these maybe
coredump-relevant code changes (and note the comments I dropped in
there when I did it):
-------------your WGET: learn snippet----------------
if (alpha < 0.000001 ) alpha = 0.000001;
if (alpha > 1.0) alpha = 1.0;
if (alpha < 0.0 ) alpha = NN_DEFAULT_ALPHA;
if (stoch_noise > 1.0) stoch_noise = 1.0;
if (stoch_noise < 0) stoch_noise = NN_DEFAULT_STOCH_NOISE;
if (soft_cycle_limit < 0)
soft_cycle_limit = NN_MAX_TRAINING_CYCLES_FROMSTART;
----------------------------------------------------------------
---------------GerH snippet-----------------
if (alpha < 0.0)
alpha = NN_DEFAULT_ALPHA; /* [i_a] moved up here by 3
lines, or it would've been useless */
if (alpha < 0.000001)
alpha = 0.000001;
if (alpha > 1.0)
alpha = 1.0;
if (stoch_noise > 1.0)
stoch_noise = 1.0;
if (stoch_noise < 0.0)
stoch_noise = NN_DEFAULT_STOCH_NOISE;
if (soft_cycle_limit < 0)
soft_cycle_limit = NN_MAX_TRAINING_CYCLES_FROMSTART;
if (internal_training_threshold < 0.0)
internal_training_threshold = NN_INTERNAL_TRAINING_THRESHOLD;
---------------------------------------------------
Furthermore, all my code uses 'double' for in-memory calculations,
while yours uses quite a few 'float's.
... I now see I lacked these two lines (not really relevant here):
------------------------
// renormalize total responses
total_icnr = total_icnr / n_classifiers;
total_ocnr = total_ocnr / n_classifiers;
-----------------------
What however MAY be important is that all your malloc()s have been
substituted with properly dimensioned calloc()s: there's another one
that could aid and abet in getting you the 'spurious behaviour' you
describe.
So summarizing:
code/results suffer/are strongly influenced by:
- random generator (and likewise: its seed).
- when calloc() 'solves' your issue, it's a strong HINT for a
particular something.
- float vs. double and floating point processors... (I've spent my
breath on int enough; this one I leave for someone else. Once is
enough.)
- memory protection actually catching an out-of-bounds access? stack
overrun kicking in?
So my suggestion is turning it around: I'd say your makefile tells you
what it is; your shell somehow still seems to allow you to run
Nagasaki code.
The fault is not the mistake, it's the messenger of something lurking
in there for some time now.
I didn't make notes when the coredumps happened to me; it was 'don't
worry, mate, it's experimental code' after all, right?
On Thu, Dec 18, 2008 at 5:14 PM, Bill Yerazunis <[email protected]> wrote:
>
> I've switched development of CRM114 over from 32-bit Fedora to 64-bit
> Ubuntu (because Ubuntu can talk to the wireless card in my new
> macbook unibody).
>
> But there's a bigger problem: when I run megatest.sh _from the
> command line_ as
>
> ./megatest.sh
>
> it behaves well. There are a few glitches where the discrepancies
> between 32-bit and 64-bit executables is more than a percent or two,
> but nothing too scarey.
>
> HOWEVER, if I "make megatest", which runs the same exact shell script
> from inside of make, I get segfaults in the neural network. Repeatedly.
>
> This is making it rather difficult to debug the failure. Any clues
> or hunches? I'm somewhat at a loss here.
>
> - Bill Yerazunis
>
> ------------------------------------------------------------------------------
> SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
> The future of the web can't happen without you. Join us at MIX09 to help
> pave the way to the Next Web now. Learn more and register at
> http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
> _______________________________________________
> Crm114-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/crm114-general
>
--
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------
------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/