Re: size of variables in hyperspace?

the reason why my training data is so large is because we have a
corpus of newspaper articles that contains about 800 million words (10
norwegian newspapers for 10 years).

in addition to the article text itself, i use all the boilerplate that
comes with it (html and javascript) when you harvest it from the url.
this, of course, makes it a lot larger.

using several different statistics files for each category does not
sound like a good idea. i already have 9 categories (plus 'uncertain'
and 'out', which are fail-categories), and i think the high number of
categories makes it harder for the classifier. we have ruthlessly
reduced the number of categories from 29 to 9.

On Tue, Jun 9, 2009 at 8:07 PM, Bill Yerazunis <[email protected]> wrote:
>
>   From: Thomas Michael Hagen <[email protected]>
>
>   thanks!
>
>   making the thickness thinner should help.
>
>   you're saying splitting the .chs-file is a good idea, but can i merge
>   them back into each other again afterwards? what tool do i use?
>
> No need to - just use all of the spoN.chs files together on the
> CLASSIFY line.
>
> Merging them together is itself problematic, because again you'd
> have a file bigger than 2^32 bytes.  Although it's possible to
> have files larger than that, right now CRM114 doesn't jump
> through the necessary hoops to do so.
>
> See:
>
>  http://en.wikipedia.org/wiki/Large_file_support
>
> for a blurb on it.
>
> To be honest, I find it astounding that you've got training
> data of more than a few tens of megabytes... which is why I
> am a bit aback about thinking about data of plural gigabytes.
>
> If I knew up front you'd be storing gigabytes, I probably would
> have designed the hyperspace data structures differently.
>
>     - Bill Yerazunis
>

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects