Re: size of variables in hyperspace?

two interesting points here:

my old-fashioned ("human brains are better than machines") colleagues
have been pushing me to use the cleaned articles (the actual text
only, not all the boilerplate). perhaps this is the time to start
doing so. on the other hand; cleaning the boilerplate out of the
articles is a non-trivial problem, and we havent' cleaned everything.

my own quick-fix for the 2Gb problem is so far to stop learning that
category when the .chs file approaches 2Gb. 'uncertain', 'domestic'
and 'sport' fill up first, with 'foreign', 'economy' and 'consumer'
not far behind. this is, of course, a rather stupid short term
solution, and i'll need some way to refine the learning without
filling up the .chs-files.

bill, a question about the hyperspace method: does it take the
ordering of features into account, or could i give it a list of all
the words in an article in no particular order and get the same
result?



On Wed, Jun 10, 2009 at 10:17 AM, Ger Hobbelt <[email protected]> wrote:
> On Tue, Jun 9, 2009 at 10:29 PM, Thomas Michael
> Hagen<[email protected]> wrote:
>>> Ouch... in fact, I'd be worried that the boilerplate around it (especially
>>> the javascript) may be including "telltales" that are dominating
>>> the decision process.
>>
>> does that mean hyperspace does not limit itself to storing the unique
>> features of each category, like osbf does?
>>
>> usually, in machine learning, it is a good idea to include as much
>> information as possible, including things that would only confuse a
>> human being...
>
> Disclaimer:
> I haven't grokked hyperspace to a level where I can be certain about
> it's internal storage patterns and algorithms, but 'reverse
> engineering' so far indicates that hyperspace stores all the features
> of each document. Of course, it's not /that/ simple, as it, while
> doing this, looks for matching feature series, i.e. feature trains (as
> in 'pulse train') which already exist in the store and 'links' to
> those.
>
> Anyway, the very abstract overview description of hyperspace would be
> this (and correct me if I'm wrong):
>
> Think of its 'universe' as ours: vacuum filled with stars. The idea
> here is to determine where your 'star under test' (== your input
> document) sits in comparison to a collection of nebulas, where one
> 'nebula' is the learned collection of 'stars' (documents) in one
> defined category. The classifier determines the distance of the
> star-under-test to each nebula and reports these findings; bluntly put
> the closest nebula 'wins' (while the pR distance to each nebula is
> reported anyway).
>
> As each document is treated as an individual 'star', the
> distance-to-nebula-X is calculated as the averaged distance to each of
> its stars (in N-dimensional space).
>
> In 3D it's a bit like attaching an elastic band between
> star/point-under-test and every star-learned, then releasing the
> star-under-test so it can freely move so that all forces equal out and
> when the jiggling has stopped, we have a look where we ended up. A
> visual evaluation determines how 'close' we are to any nebula in that
> space and which one is the 'closest'.
> (Of course, since it's all more sophisticated than this, theoretically
> we're not playing in 3D but N-dimensional space, where is a high
> number about equal to the number of independent features in there.
> Anyway, the important bit is that each /document/ plays a significant
> part here, contrary to the Bayesians, where the 'summary/essence of
> trained documents' is used -- where 'essence' is defined as the
> 'essence' distillable from documents from a /bayesian/ perspective. As
> Hyperspace is not bayesian, it can 'see' things a bayesian classifier
> would not. The other side of the coin is the algorithm requires each
> star to remain [somewhat] identifiable until eternity -->
> ever-growing, non-limited CSS store format.)
>
> Given this (and assuming I hit the nail at least ball-park) it makes
> 'sense' to treat hyperspace training needs different from bayesians:
> if you know a way to distill 'stars' (== documents) down to their
> 'essence' (here, in the sense that you 'know' how you want Hyperspace
> to evaluate/consider all those stars, ever), you should preprocess the
> documents before featurizing them. The 'how' in that last sentence is
> the shortest word for a lot of voodoo to get thing working 'better'.
> Voodoo == test every idea, that makes sense to you, and then test a
> few that don't, as well.
>
>
> (PS: given that assertion you reported before, sounds to me like you
> are using a GerH build. Well, good to know those assertions pay off,
> even though they do not really solve your issue.
> I'm not 100% sure, because I haven't gone past the 2GB-per-CSS barrier
> yet, but does building and running this stuff on a native 64-bit
> platform (and crm114 as a native 64-bit app) give you this bother
> too?)
>
>
>
> (PPS: I didn't mention run-time performance, but given the global
> descriptions, rough guestimates for Bayesians would be O(N) where N is
> the number of features in the document-under-test (as those match
> using a ~ O(1) hash table approach, with one hash table per category),
> while Hyperspace would be more like O(N*M) where M is the total number
> of trained documents over all categories, as Hyperspace uses something
> /somewhat/ akin to a skip-list, one per category. The sharp end of the
> point would be that Hyperspace is slower than bayesians when training
> counts increase. Well, you have probably noticed this because you are
> using large CSS stores, correct?)
>
>
>
> PPPS: if you need to cross the 2GB barrier into TB space, you might
> want to have a software dev look at the file I/O and classifier code;
> this is one place where 'long' is 'better' than 'int' but ***only***
> for LP64 platforms, i.e. BSD/Linux(with GCC), ***NOT*** 64-bit
> Windows, as that one is LLP64.) A _portable_ software fix means
> pervasive use of the 'size_t' and (on some platforms non-existing,
> though definable) 'ssite_t' types for anything file/data store
> related.
> The bloody 'quick fix' is to compile and run this on an ILP64
> platform, e.g. Alpha + Digital UNIX (or other high-end UNIXes; Linux
> unfortunately is LP64 as long as you stick with GCC. Unless they've
> got an option that says 'treat type 'int' as 64-bit' in their latest
> compiler...
> The hackiest-ever 'quick fix' for this is to
> #define int long
> in config.h for the entire crm114 sourcecode collection and compile it
> on a LP64/GCC Linux box (and pray).
>
> I mention those 'quick fixes' as the fast escape routes as I haven't
> got the time to thoroughly upgrade the code to give you Terabyte
> capability right now, sorry.
>
>
>
> All of which (but the dangerously hacky bits) doesn't 'solve' the 2GB
> barrier issue, but alas, a little downsizing here or there might help
> you for now.
>
>
>
>
> --
> Met vriendelijke groeten / Best regards,
>
> Ger Hobbelt
>
> --------------------------------------------------
> web:    http://www.hobbelt.com/
>        http://www.hebbut.net/
> mail:   [email protected]
> mobile: +31-6-11 120 978
> --------------------------------------------------
>

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects