Re: size of variables in hyperspace?

thanks!

making the thickness thinner should help.

you're saying splitting the .chs-file is a good idea, but can i merge
them back into each other again afterwards? what tool do i use?



On Tue, Jun 9, 2009 at 4:28 PM, Bill Yerazunis <[email protected]> wrote:
>
>   From: Thomas Michael Hagen <[email protected]>
>
>   it could seem that crm thinks the .chs-file is empty when it reaches a
>   certain size. i'm now getting the same error on spo.chs, which
>   contains sports. that file reaches about 2Gb in size after twenty
>   minutes of learning.
>
> OUCH.
>
> 2 Gb is a magical place - it's where the file size wraps around a
> 32-bit INT.
>
> Thus, the file would be of negative size, and you get problems.
>
> This is going to be difficult to fix, seeing as we're trying to get
> away from "long" because "long" changes size depending on platform.
>
> I take it you're running on an X64 machine, otherwise you'd have
> run out of mmap memory long ago.
>
> As a temporary workaround, try creating file "spo1.chs", "spo2.chs"
> and so on as each file gets past a gigabyte or so.
>
> I am impressed that you have that much data.  Given the large
> data volume, you might also want to try making the thickness thinner
> to minimize the amount of data that gets put into your .chs files.
>
>   - Bill Yerazunis
>
>
>   On Tue, Jun 9, 2009 at 9:59 AM, Thomas Michael Hagen
>   <[email protected]> wrote:
>   > i'm playing with hyperspace, since it seems osbf won't converge on any
>   > sort of learning on my 9 + 2 categories of newspaper articles.
>   >
>   > hyperspace seemed very promising until i got this error:
>   >
>   > crm114: *UNTRAPPABLE ERROR*
>   > Better start screaming, guv', since the software's just gone critical:
>   > assertion 'feature_pos + single_featureblock_size <= features_bufferlen' failed!
>   > Sorry, but this program is very sick and probably should be killed off.
>   > This happened at line 416 of file
>   > /sandisk/prosjekt/aviskorpus/emneklassifisering/thomas/hyperspace/learn-or-compare.crm:
>   >    learn <hyperspace> [:my-args:] (unc.chs)
>   > (runtime system location: crm_vector_tokenize.c(753) in routine:
>   > crm_vector_tokenize)
>   > [st08764@gandalf hyperspace]$
>   >
>   >
>   > i'm not sure what this means, but i'm guessing it has something to do
>   > with the size of the input variable?
>   >
>   > i start the program with:
>   >
>   > /usr/local/crm114/bin/crm114 -w 152500000 learn-or-compare.crm dag-filer
>   >
>   > where dag-filer is a list of files containing anything from 0 to
>   > several hundred articles.
>   >
>   > the 'unc.chs' file referred to in the error message is the 'uncertain'
>   > category, which means a classification attempt failed.
>   >
>   > the 'learn-or-compare.crm' program learns a number of articles, then
>   > tries to classify, and compares its results to a gold standard.
>   >
>   > so, what sizes should i change to get rid of this problem?
>   >
>
>   ------------------------------------------------------------------------------
>   Crystal Reports - New Free Runtime and 30 Day Trial
>   Check out the new simplified licensing option that enables unlimited
>   royalty-free distribution of the report engine for externally facing
>   server and web deployment.
>   http://p.sf.net/sfu/businessobjects
>   _______________________________________________
>   Crm114-general mailing list
>   [email protected]
>   https://lists.sourceforge.net/lists/listinfo/crm114-general
>
>

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects