Re: Crm114-general Digest, Vol 33, Issue 12

   From: Thomas Michael Hagen <[email protected]>


   the thing is, i am not filtering spam; i'm doing a topic classifier (with 29
   categories) as a research project, so there is a lot of academic curiosity
   as to what features - and what tokens - are in there.

   it would be nice to have two things: first, a utility that would output the
   features and tokens from a .cfc-file without getting a vasectomy.
   second, a variable that contained the unique features that a particular
   sample generated (perhaps these could be included in the :stats:-variable).

   i wonder if is this something that bill has considered, or if  it is
   completely undoable.

It's trivial. In fact, if memory serves correctly, it's already there
for all of the vector-tokenizing classifiers and might be there for
all of them (and in the next release, all classifiers are vector
tokenizing, so it will be there)

Turn on tracing with "-t" on the command line.  That will
give you a _massive_ data dump on stderr of everything that happened.

Among all of that dump, you will see a bunch of lines that look like

  Token; k: 0 T.O: 301 len 5 ( 1 6 on >have.< )

Write yourself a little CRM114 script that finds things between the >
and < and you have all of the tokens.

Now, depending on how you set up your classifier (i.e. Unigram or
not, STRING or not, special pipeline or such) you will need to apply
that to your token stream to get the features that actually got
used.

You can then go wild on the statistical analysis.  

(note- it's NOT easy to get the post-pipeline feature strings, 
because in VT we get tokens, then hash, then apply the pipeline 
coefficients and so the feature strings are never actually computed)

	     - Bill Yerazunis

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H