Re: Sorting many small data sets

   From: Paolo <[email protected]>

   On Sun, Dec 14, 2008 at 02:00:37PM -0700, Chris Babcock wrote:
   ...
   > Each record is only a about a dozen fields (~250 bytes each). There are
   > about 15,000 records distributed over a dozen or so flat file
   ...
   > for this Pr or better.'" Intuitively, hyperspace seems to be the most
   > likely candidate because of the speed of learning and the small disk
   > foot print. Each corpus will be very small, the documents small and
   > fairly similar to begin with, and there will be 15,000+ classification
   > files floating around at the end of it, which would be nice to keep for
   > reducing future duplications. Is there a reasonable chance of this
   > approach working given the obvious weaknesses in the quality of the
   > data? 

   choice of HS seems fine - works better with small amount of data, _growing_
   cssfile size - I'd rather say the other way around: you've got a nice 
   multiclass case, pls go ahead, try and report back how it goes ;)

Yep, I agree.  Hyperspace is nice and small on the disk.  Give that
a try first.

Another (currently nonexistent) algorithm is to use a stupidified
FSCM.  The new FSCM uses all sorts of hacks for speed with large data
sources... and those lookaside tables take up a lot of space.  A
"dumb" FSCM (that uses LZ77 only with (say) a 1 to 10-megabyte window)
might actually work better for you.

Anybody want to code one up?  You can base it right off of the current
FSCM source module.  We can put it into the system as the "LZ77" classifier.

  - Bill Yerazunis

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/