Re: Crm114-general Digest, Vol 33, Issue 12

thanks!

all right, i'll try that, and let you know how it works out.


On Thu, Feb 19, 2009 at 2:55 PM, Bill Yerazunis <[email protected]> wrote:

>
>   From: Thomas Michael Hagen <[email protected]>
>
>
>    the thing is, i am not filtering spam; i'm doing a topic classifier
> (with 29
>   categories) as a research project, so there is a lot of academic
> curiosity
>   as to what features - and what tokens - are in there.
>
>   it would be nice to have two things: first, a utility that would output
> the
>   features and tokens from a .cfc-file without getting a vasectomy.
>   second, a variable that contained the unique features that a particular
>   sample generated (perhaps these could be included in the
> :stats:-variable).
>
>   i wonder if is this something that bill has considered, or if  it is
>   completely undoable.
>
> It's trivial. In fact, if memory serves correctly, it's already there
> for all of the vector-tokenizing classifiers and might be there for
> all of them (and in the next release, all classifiers are vector
> tokenizing, so it will be there)
>
> Turn on tracing with "-t" on the command line.  That will
> give you a _massive_ data dump on stderr of everything that happened.
>
> Among all of that dump, you will see a bunch of lines that look like
>
>  Token; k: 0 T.O: 301 len 5 ( 1 6 on >have.< )
>
> Write yourself a little CRM114 script that finds things between the >
> and < and you have all of the tokens.
>
> Now, depending on how you set up your classifier (i.e. Unigram or
> not, STRING or not, special pipeline or such) you will need to apply
> that to your token stream to get the features that actually got
> used.
>
> You can then go wild on the statistical analysis.
>
> (note- it's NOT easy to get the post-pipeline feature strings,
> because in VT we get tokens, then hash, then apply the pipeline
> coefficients and so the feature strings are never actually computed)
>
>             - Bill Yerazunis
>

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general