Re: Behavior divergence in makefile vs. "in person".

   From: "Ger Hobbelt" <[email protected]>


   Welcome to the 64-bit club. ;-)

Yeah, but I now need to somehow have a 32-bit machine "on tap"
to test the 32-bit builds as well as the 64-bit builds.

It's very hard to test and debug if you don't have either the
hardware or a bit-accurate simulation of the hardware.  You can't
effectively work a bug you can't reproduce.

   Haven't checked lately, but NN has most often been this flaky for me
   (my current code here 'works' (as in: does not crash), but that's no
   help as I replaced the random generator to ensure cross-system
   reproducible results with CRM114 -- after all, GerH builds have
   automated reference comparison checks for each of the tests, and NN is
   very sensitive to different random gens: each system has it's own.
   I recall the many times NN barfed out on me fatally, as now happens to
   you. Whenever this happened, both commandline and make runs produced
   the same fatal result.

That is the wierdness.  Why should something work right under 
command line, and fail inside Make?  Oh- it does fail under Valgrind
too... that's good.

   One thing I can imagine is a changed stack limit or other 'tighter'
   memory protection settings inside make. Note though that I strongly
   suspect your /commandline/ from not showing what is actually wrong;
   for instance, it's already for a VERY long time that
       -{learn < neural refute fromstart > (q_test.css) /[[:graph:]]+/}  ...
   blows up over here. And when it did, it was often system dependent
   here (Win32 / Win64 / SuSE64). This was mentioned before when I was
   working on crm114 intensively half a year or more ago.

Yeah.  The problem is that with the prior hardware, there was no
way I could run even a VM.  I barely had disk space for my (legal!)
MP3's and some test spam.  

That's changed now; I might be able to have two or three VMs 
around at once.  :)

   I just now quickly checked my own tests/Makefile.am and grep for
   CRM114_CHECK_OVERRIDE=0 in there hints:
   that's equivalent to the third NN line in your megatest.sh coredumping
   and continue on down from there, as I experienced it. Of course, those
   two learns blowing a gasket leads to disaster for subsequent classify,
   so the next chance at a non-crashing trial is test #2:


   # **** Alternate Neural Network Classifier test script
   **********************************************************
   Neural_Network_test2: $(TEST_PREREQUISITES)
   test_timing_minutes_announcement alternating_example_neural.crm
	   $(E) "*******************************************"
	   $(E) "* Alternate Neural Network Classifier test script "
	   $(E) "*******************************************"
	   -$(SILENT)rm -f i_test.css
	   -$(SILENT)rm -f q_test.css
	   $(SILENT)\
	   CRM114_CHECK_FILTER_ARGS="-v prob=0.2 -v pR=10.0"; export
   CRM114_CHECK_FILTER_ARGS; \
	   ./testscript.sh alternating_example_neural.crm "" -
   "$(builddir)/crm114_classify.filter.sh"


   which tells us that, for GerH crm114, this one is within +/- 10.0
   (probably about 30% of that range: +/-3) range of your megatest.sh
   reference results. And does NOT crash (or it'd have had another
   CRM114_CHECK_OVERRIDE=0 as well in there. The other ('older') NN test
   fails dramatically often enough, like you mention now.


   I checked, and currently test #1 doesn't dump core like yours, but the
   latest WGET for NN I see is from around summer this year, so I am not
   sure we're anywhere near to 'in sync'. Relatively speaking.


   On the one hand I feel your pain in tracking this bastard down, on the
   other hand I am somehow glad you hit these: just maybe, finally these
   pieces of code will get the attention they deserve.

Again, if I could have tickled the bugs, I'd have worked them.

The bug in FSCM was the same way- every test _I_ ran was fine;
nice clean results.  I only got a hint of it when I got email
that there was trouble, and I spent more time trying to 
reproduce the bug than the actual fix took.  (it took a long
time; the only reason I kept at it was because I knew the person
involved and helped set them up and so I knew it wasn't "cockpit
error".)


   Anyway, let's see if we can get you anywhere near a 'working' system
   again. Here's the main (important) differences between GerH and your
   wget:

   1) I replaced the random generator - and NN is VERY sensitive to that
   one. Each system has its own and that's making matters very bad for
   automated testing setups like mine: GerH 'make tests' actually
   verifies the tests and those differences in rand() on the various
   boxes leads to WILDLY varying NN output. Currently mine says:

Sorry, but that's just attacking the symptom, not the cause.

Failing to converge is always possible with a NN; it's part of
the way they work.  

SEGFAULTING, on the other hand, should be impossible, no matter
what the initial conditions set up by the random number generator.


   -------------snippet-----------
   TEST SCRIPT + COMMANDLINE:
   -{learn < neural refute fromstart > (i_test.css) /[[:graph:]]+/}

   RETURN CODE:
   0

   STDERR TEST OUTPUT:

   crm114: *WARNING*
    neural: failed to converge within the training limit.   Beware your results.
    You might want to consider a larger network as well.
   I'll try to keep working.
   This happened at line 2 of file (from command line):
       learn < neural refute fromstart > (i_test.css) /[[:graph:]]+/
   (runtime system location: crm_neural_net.c(2208) in routine:
   crm_neural_net_learn)

   STDOUT TEST OUTPUT:

   TEST SCRIPT + COMMANDLINE:
   -{ isolate (:s:); {classify < neural > ( i_test.css | q_test.css )
   (:s:)/[[:graph:]]+/ ; output / type I :*:_nl::*:s::*:_nl:/} alius {
   output / ty
   pe Q :*:_nl::*:s::*:_nl:/ } }

   RETURN CODE:
   0

   STDERR TEST OUTPUT:

   STDOUT TEST OUTPUT:
    type Q
   CLASSIFY fails; success probability: 0.227734  pR: -76.3110
   Best match to file #1 (q_test.css) prob: 0.7723  pR: 76.3110
   Total features in input file: 268
   #0 (i_test.css): icnr: 0.17 ocnr: 0.64 prob: 2.28e-01, pR: -38.34
   #1 (q_test.css): icnr: 0.88 ocnr: 0.08 prob: 7.72e-01, pR:  76.31
   -----------------------------------

   Jack in another seed or another high quality RNG (or a low quality
   one, whatever) and that warning may go away, the coredump can rear its
   head again for the refute-learn lines (and onwards of course), etc.
   etc. Options galore.

The entropy for the random input should not affect the correctness
of the code.  At least, it BETTER NOT.  That's a bug of the first
water (though there's no guarantee that a NN will actually converge;
maybe one should think of the whole idea of NN's as "defective" in
that scheme of thinking).


   2) a quick diff between wget and GerH shows these maybe
   coredump-relevant code changes (and note the comments I dropped in
   there when I did it):

   -------------your WGET: learn snippet----------------
       if (alpha < 0.000001 ) alpha = 0.000001;
       if (alpha > 1.0) alpha = 1.0;
       if (alpha < 0.0 ) alpha = NN_DEFAULT_ALPHA;
       if (stoch_noise > 1.0) stoch_noise = 1.0;
       if (stoch_noise < 0) stoch_noise = NN_DEFAULT_STOCH_NOISE;
       if (soft_cycle_limit < 0)
	 soft_cycle_limit = NN_MAX_TRAINING_CYCLES_FROMSTART;
   ----------------------------------------------------------------

   ---------------GerH snippet-----------------
	       if (alpha < 0.0)
		   alpha = NN_DEFAULT_ALPHA; /* [i_a] moved up here by 3
   lines, or it would've been useless */
	       if (alpha < 0.000001)
		   alpha = 0.000001;
	       if (alpha > 1.0)
		   alpha = 1.0;
	       if (stoch_noise > 1.0)
		   stoch_noise = 1.0;
	       if (stoch_noise < 0.0)
		   stoch_noise = NN_DEFAULT_STOCH_NOISE;
	       if (soft_cycle_limit < 0)
		   soft_cycle_limit = NN_MAX_TRAINING_CYCLES_FROMSTART;
	       if (internal_training_threshold < 0.0)
		   internal_training_threshold = NN_INTERNAL_TRAINING_THRESHOLD;
   ---------------------------------------------------

Your code is correct.


   Furthermore, all my code uses 'double' for in-memory calculations,
   while yours uses quite a few 'float's.

Shouldn't cause a segfault though.  _Better_ _not_, at least.
	  	

		- Bill Yerazunis

------------------------------------------------------------------------------