Re: size of variables in hyperspace?

   the reason why my training data is so large is because we have a
   corpus of newspaper articles that contains about 800 million words (10
   norwegian newspapers for 10 years).

   in addition to the article text itself, i use all the boilerplate that
   comes with it (html and javascript) when you harvest it from the url.
   this, of course, makes it a lot larger.

Ouch... in fact, I'd be worried that the boilerplate around it (especially
the javascript) may be including "telltales" that are dominating
the decision process.

I think you might want to strip that stuff out.  Anything that
a human looking at the screen wouldn't see as text is a candidate
for removal.

   using several different statistics files for each category does not
   sound like a good idea. i already have 9 categories (plus 'uncertain'
   and 'out', which are fail-categories), and i think the high number of
   categories makes it harder for the classifier. we have ruthlessly
   reduced the number of categories from 29 to 9.

Maybe things would work better if you got rid of the boilerplate.

It's a worthwhile experiment.

   - Bill

   On Tue, Jun 9, 2009 at 8:07 PM, Bill Yerazunis <[email protected]> wrote:
   >
   >   From: Thomas Michael Hagen <[email protected]>
   >
   >   thanks!
   >
   >   making the thickness thinner should help.
   >
   >   you're saying splitting the .chs-file is a good idea, but can i merge
   >   them back into each other again afterwards? what tool do i use?
   >
   > No need to - just use all of the spoN.chs files together on the
   > CLASSIFY line.
   >
   > Merging them together is itself problematic, because again you'd
   > have a file bigger than 2^32 bytes.  Although it's possible to
   > have files larger than that, right now CRM114 doesn't jump
   > through the necessary hoops to do so.
   >
   > See:
   >
   >  http://en.wikipedia.org/wiki/Large_file_support
   >
   > for a blurb on it.
   >
   > To be honest, I find it astounding that you've got training
   > data of more than a few tens of megabytes... which is why I
   > am a bit aback about thinking about data of plural gigabytes.
   >
   > If I knew up front you'd be storing gigabytes, I probably would
   > have designed the hyperspace data structures differently.
   >
   >     - Bill Yerazunis
   >


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general