Re: size of variables in hyperspace?

   From: Thomas Michael Hagen <[email protected]>

   it could seem that crm thinks the .chs-file is empty when it reaches a
   certain size. i'm now getting the same error on spo.chs, which
   contains sports. that file reaches about 2Gb in size after twenty
   minutes of learning.

OUCH.

2 Gb is a magical place - it's where the file size wraps around a
32-bit INT.  

Thus, the file would be of negative size, and you get problems.

This is going to be difficult to fix, seeing as we're trying to get
away from "long" because "long" changes size depending on platform.

I take it you're running on an X64 machine, otherwise you'd have
run out of mmap memory long ago.

As a temporary workaround, try creating file "spo1.chs", "spo2.chs"
and so on as each file gets past a gigabyte or so.  

I am impressed that you have that much data.  Given the large
data volume, you might also want to try making the thickness thinner
to minimize the amount of data that gets put into your .chs files.

   - Bill Yerazunis


   On Tue, Jun 9, 2009 at 9:59 AM, Thomas Michael Hagen
   <[email protected]> wrote:
   > i'm playing with hyperspace, since it seems osbf won't converge on any
   > sort of learning on my 9 + 2 categories of newspaper articles.
   >
   > hyperspace seemed very promising until i got this error:
   >
   > crm114: *UNTRAPPABLE ERROR*
   > Better start screaming, guv', since the software's just gone critical:
   > assertion 'feature_pos + single_featureblock_size <= features_bufferlen' failed!
   > Sorry, but this program is very sick and probably should be killed off.
   > This happened at line 416 of file
   > /sandisk/prosjekt/aviskorpus/emneklassifisering/thomas/hyperspace/learn-or-compare.crm:
   >    learn <hyperspace> [:my-args:] (unc.chs)
   > (runtime system location: crm_vector_tokenize.c(753) in routine:
   > crm_vector_tokenize)
   > [st08764@gandalf hyperspace]$
   >
   >
   > i'm not sure what this means, but i'm guessing it has something to do
   > with the size of the input variable?
   >
   > i start the program with:
   >
   > /usr/local/crm114/bin/crm114 -w 152500000 learn-or-compare.crm dag-filer
   >
   > where dag-filer is a list of files containing anything from 0 to
   > several hundred articles.
   >
   > the 'unc.chs' file referred to in the error message is the 'uncertain'
   > category, which means a classification attempt failed.
   >
   > the 'learn-or-compare.crm' program learns a number of articles, then
   > tries to classify, and compares its results to a gold standard.
   >
   > so, what sizes should i change to get rid of this problem?
   >

   ------------------------------------------------------------------------------
   Crystal Reports - New Free Runtime and 30 Day Trial
   Check out the new simplified licensing option that enables unlimited
   royalty-free distribution of the report engine for externally facing 
   server and web deployment.
   http://p.sf.net/sfu/businessobjects
   _______________________________________________
   Crm114-general mailing list
   [email protected]
   https://lists.sourceforge.net/lists/listinfo/crm114-general


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general