Re: Case sensitivity in CLASSIFY and LEARN

It appears that CRM114 is not able to trap this particular error where a variable is too big for the isolated data area.  To test the trap statement I ran the following and saw the output fine.

{
    window
    {
         input (:fifo:) [nonexistentfile]
    }
    trap /.*/
    {
         output /error caught\n/
    }
}

But if I provide the name of a large file (16M) that actually exists instead the program exits without any output.  Any idea what's going on here?  I'm currently running 20070810-BlameTheSegfault.  Thanks.

-James



----- Original Message ----
From: Bill Yerazunis <[email protected]>
To: James Lee <csejl-/[email protected]>
Cc: [email protected]
Sent: Tuesday, September 30, 2008 6:38:47 AM
Subject: Re: [Crm114-general] Case sensitivity in CLASSIFY and LEARN


   From: James Lee <csejl-/[email protected]>
   Hi,

(sorry for the delay, I was on vacation)

   I'm wondering if the CLASSIFY and LEARN statements are case
   sensitive.  If so, does that mean I have to convert all texts into
   the same case before feeding them into the CLASSIFY and LEARN
   statements to achieve higher accuracy (without training with all
   possible combinations of cases for each word)?  It appears that
   there's no flag to turn on/off case sensitivity for these two
   statements.  The <nocase> flag only applies to the word pattern
   part of the statement, correct?

Correct... and unless you have really good reason to believe that
character case in your incoming texts is _really_ meaningless, then
I'd strongly suggest leaving the documents case-sensitive.

However, if you _know_ the case is misleading, then a simple TRANSLATE
will fix it.

Just insert a character translation statement like this:

   translate [:source_var:] (:result_var:) /A-Z/ /a-z/

which will translate all upper-case letters to their lower case
equivalents (note- this is only for LATIN-1 chars; accented chars are
unaffected).


   In my daemonization code (using FIFO), if I feed in a buffer that's
   too big I get the following error and the entire program exits:

   /usr/bin/crm: *ERROR* This program has overflowed the ISOLATEd data
    area with a variable that's just too big.  The bad variable was
    named: :fifo: Sorry, but this program is very sick and probably
    should be killed off.  This happened at line 12 of file
    /home/me/daemon.crm The line was: --> input (:fifo:) [
    \/tmp\/dynamic_fifo ] <--

   I used the -w flag to increase the max data window size to get
   around this problem.  Also, I plan on limiting the size of the
   buffer on the client side to reduce the likelihood of this
   happening.  My question is: Is there a way to prevent the entire
   program from exiting in case something like this happens?  For
   example, is it possible to ignore this error, re-initialize and
   carry on (to the next iteration of a loop)?

Yes indeed!

This is exactly what the TRAP statement does.  And- it sounds
like your fifo is not getting emptied.  Remember, WINDOW doesn't 
shorten the source variable.

Basically, a TRAP looks at the error message with a regex, and if the
regex matches, then the code inside the TRAP executes.  That code can
do anything it wants (including cause other TRAPs).  In your case, you
might just want to erase all of the big vars and start again from
zero; for anti-DoS reasons, a running CRM114 program _cannot_ increase
it's own window size (although it can SYSCALL another copy of itself
with a bigger window.)

Here's a skeleton.  Note that a TRAP needs a { } wrapper around the
code it's trapping.

    {
      {
    My_code_goes_here.
          Maybe_it_fails.
    Blah_Blah_Blah...
      }
      trap /.*/    ###<---   this trap catches anything
      {
        Fixup_code_goes_here.
        For_Example
        isolate (:fifo:) //
      }
    } 

Another example is the file "traptest.crm" which is one of the 
megatest tests, and comes free in your CRM114 kit.

   - Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/