Re: hyperspace vs. hyperspace + unique

On Thu, Nov 5, 2009 at 12:15 AM, Tomas <[email protected]> wrote:
> Greetings,
>
> i didnt find enough materials about mythology behind Unique with Hyperspace.

LOL :-) Nice play of words! Yes, I agree. Sometimes I get the feeling
there's an abundance of mythology around crm114 (and statistical
filters in general) too.


> Whats expected impact of Unique on hyperspace in spam filtering ??
>
> anyone ?

<unique> as a rule discards any feature hashes which have occurred
previously within the same input, so <wiggles hand> I'd say <unqiue>,
for any classifier, makes documents with repetitive parts in them
[almost] identical to documents without the repetitions.

Take example doc1 and doc2:

doc1: 'a b a b c'
doc2: 'a b c'

where at a 4-wide stride VT (vector tokenizer) action, the following
feature hashes would be produced for doc1 (where X is the special
DEADBEEF hash which represents a kind of 'null' word):

doc1:

a b
a a
a b
a c

b a
b b
b c
b X

a b
a c
a X
a X

b c
b X
b X
b X

c X
c X
c X
c X


doc2:

a b
a c
a X
a X

b c
b X
b X
b X

c X
c X
c X
c X


doc1 with unique:

a b
a a
----
a c

b a
b b
b c
b X

----
----
a X
----

----
----
----
----

c X
----
----
----


doc2 with unique:

a b
a c
a X
----

b c
b X
----
----

c X
----
----
----


so one might say that doc1 and doc2 are almost the same when both get
fed to a <unique> flagged classifier (that is us ignoring those
features

a a
b a
b b
b c

there in doc1, yes, but imagine the repetitive parts to be larger and
the inputs to be larger than 5 words too so one might argue that the
difference is negligible.

The added ;benefit' of 'unique' is that repetitive strips in the input
don't weight in more than the non-repetitive parts: 'a b' is counted
only once, no matter how many 'a b' strips you concatenate in the
input.



Hope that cleared a bit of the clouds.


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july