Re: size of variables in hyperspace?

On Wed, 10 Jun 2009 10:30:56 +0200
Thomas Michael Hagen <[email protected]> wrote:

> my own quick-fix for the 2Gb problem is so far to stop learning that
> category when the .chs file approaches 2Gb. 'uncertain', 'domestic'
> and 'sport' fill up first, with 'foreign', 'economy' and 'consumer'
> not far behind. this is, of course, a rather stupid short term
> solution, and i'll need some way to refine the learning without
> filling up the .chs-files.

This isn't as stupid as it may appear. You're emulating a natural
strategy for information overload. Admittedly the barrier is an
artificial one, but the strategy is acceptable... providing, of course,
that it works acceptably well.

The next step would be to set up sub-categories when categories fill.
A script that could monitor its own storage, recognize when to stop
training a category, re-analyze its corpus in that category and define
relevant sub-categories would be very powerful. There's no "out of the
box" way to do that with CRM, but it might be worthwhile to do manually
until you can develop a sufficient handle on something like the Python
Natural Language Toolkit (NLTK) or duplicate the relevant functions in
CRM.

Also, if a category is one that evolves then you might employ a
'mentoring' strategy. When a feature file matures, use feedback from a
classify based on that file to train a replacement. Once the
replacement has been trained to an acceptable level, it can be used
instead of the older file.

Chris

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQEcBAEBAgAGBQJKL/OLAAoJEASgqNsqZfCHzw8IAIA97lViDtI++9gCX7nhNNQF
EkhrbDc2R8XB1K/6LgR6MJNlntjginWix6xg9gKkE1Iy4aN2nG19hPZBXOyyOgA4
EwM6naLLrdwpJMPsnP3IZ1YqmPRp3drQocrDGJzA7nSwmi3hHOj2MmnJbpFp8uVO
Dr/NsHkf9MZmij7diZ/gZOLj+IAcLjBxj+CIveevVYUF86P8rOXPVbL1i0Z4ZdSI
F6cnvhPR43ApwdjcDQf64IJ0Z6ejjg3Tu465ZQ9HRVEkAvXt/HNNauBDs7g1oyAz
MU/SAZzUkmFQbDUUCCuiSugXZzskpOnBQOKKL509nLHOoAgW1ip0E1V1yovV/C8=
=TC78
-----END PGP SIGNATURE-----