Re: size of variables in hyperspace?

On Tue, Jun 9, 2009 at 10:29 PM, Thomas Michael
Hagen<[email protected]> wrote:
>> Ouch... in fact, I'd be worried that the boilerplate around it (especially
>> the javascript) may be including "telltales" that are dominating
>> the decision process.
>
> does that mean hyperspace does not limit itself to storing the unique
> features of each category, like osbf does?
>
> usually, in machine learning, it is a good idea to include as much
> information as possible, including things that would only confuse a
> human being...

Disclaimer:
I haven't grokked hyperspace to a level where I can be certain about
it's internal storage patterns and algorithms, but 'reverse
engineering' so far indicates that hyperspace stores all the features
of each document. Of course, it's not /that/ simple, as it, while
doing this, looks for matching feature series, i.e. feature trains (as
in 'pulse train') which already exist in the store and 'links' to
those.

Anyway, the very abstract overview description of hyperspace would be
this (and correct me if I'm wrong):

Think of its 'universe' as ours: vacuum filled with stars. The idea
here is to determine where your 'star under test' (== your input
document) sits in comparison to a collection of nebulas, where one
'nebula' is the learned collection of 'stars' (documents) in one
defined category. The classifier determines the distance of the
star-under-test to each nebula and reports these findings; bluntly put
the closest nebula 'wins' (while the pR distance to each nebula is
reported anyway).

As each document is treated as an individual 'star', the
distance-to-nebula-X is calculated as the averaged distance to each of
its stars (in N-dimensional space).

In 3D it's a bit like attaching an elastic band between
star/point-under-test and every star-learned, then releasing the
star-under-test so it can freely move so that all forces equal out and
when the jiggling has stopped, we have a look where we ended up. A
visual evaluation determines how 'close' we are to any nebula in that
space and which one is the 'closest'.
(Of course, since it's all more sophisticated than this, theoretically
we're not playing in 3D but N-dimensional space, where is a high
number about equal to the number of independent features in there.
Anyway, the important bit is that each /document/ plays a significant
part here, contrary to the Bayesians, where the 'summary/essence of
trained documents' is used -- where 'essence' is defined as the
'essence' distillable from documents from a /bayesian/ perspective. As
Hyperspace is not bayesian, it can 'see' things a bayesian classifier
would not. The other side of the coin is the algorithm requires each
star to remain [somewhat] identifiable until eternity -->
ever-growing, non-limited CSS store format.)

Given this (and assuming I hit the nail at least ball-park) it makes
'sense' to treat hyperspace training needs different from bayesians:
if you know a way to distill 'stars' (== documents) down to their
'essence' (here, in the sense that you 'know' how you want Hyperspace
to evaluate/consider all those stars, ever), you should preprocess the
documents before featurizing them. The 'how' in that last sentence is
the shortest word for a lot of voodoo to get thing working 'better'.
Voodoo == test every idea, that makes sense to you, and then test a
few that don't, as well.


(PS: given that assertion you reported before, sounds to me like you
are using a GerH build. Well, good to know those assertions pay off,
even though they do not really solve your issue.
I'm not 100% sure, because I haven't gone past the 2GB-per-CSS barrier
yet, but does building and running this stuff on a native 64-bit
platform (and crm114 as a native 64-bit app) give you this bother
too?)



(PPS: I didn't mention run-time performance, but given the global
descriptions, rough guestimates for Bayesians would be O(N) where N is
the number of features in the document-under-test (as those match
using a ~ O(1) hash table approach, with one hash table per category),
while Hyperspace would be more like O(N*M) where M is the total number
of trained documents over all categories, as Hyperspace uses something
/somewhat/ akin to a skip-list, one per category. The sharp end of the
point would be that Hyperspace is slower than bayesians when training
counts increase. Well, you have probably noticed this because you are
using large CSS stores, correct?)



PPPS: if you need to cross the 2GB barrier into TB space, you might
want to have a software dev look at the file I/O and classifier code;
this is one place where 'long' is 'better' than 'int' but ***only***
for LP64 platforms, i.e. BSD/Linux(with GCC), ***NOT*** 64-bit
Windows, as that one is LLP64.) A _portable_ software fix means
pervasive use of the 'size_t' and (on some platforms non-existing,
though definable) 'ssite_t' types for anything file/data store
related.
The bloody 'quick fix' is to compile and run this on an ILP64
platform, e.g. Alpha + Digital UNIX (or other high-end UNIXes; Linux
unfortunately is LP64 as long as you stick with GCC. Unless they've
got an option that says 'treat type 'int' as 64-bit' in their latest
compiler...
The hackiest-ever 'quick fix' for this is to
#define int long
in config.h for the entire crm114 sourcecode collection and compile it
on a LP64/GCC Linux box (and pray).

I mention those 'quick fixes' as the fast escape routes as I haven't
got the time to thoroughly upgrade the code to give you Terabyte
capability right now, sorry.



All of which (but the dangerously hacky bits) doesn't 'solve' the 2GB
barrier issue, but alas, a little downsizing here or there might help
you for now.




-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing 
server and web deployment.
http://p.sf.net/sfu/businessobjects