Re: Any way to tell what features contributed the most for a particular decision?

On Thu, Oct 16, 2008 at 7:45 PM, Robert Swirsky-Warner
<[email protected]> wrote:
> Thanks! That's what I was afraid of. I was hoping that I could get something
> out of the Bayesean/Markov ones.
>
> The challenge is, when using this for a client, they'd like to see reports
> of the most significant features. I may try to build a "debug" data
> structure for the hyperspace classifier that holds more information.

Heh. One important thing missing in there (important for the way to
tackle this; would be doable for OSB/OSBF/Markovian/Winnow; Hyperspace
_might_ be doable but that would be a real brain cruncher for me and
the ones beyond are currently /definitely/ beyond my reach, which
might give you a bit of a hunch regarding viability for yourself). The
thing missing:

do you want ot give them a report of 'most significant items per
classified document' or 'most significant items overall'?

In case of the first, I suggest you take a look at the GerH
experimentals and the

    crm_analysis_mark(..)

trace calls in there plus maybe a quick glance at the extra (/very/
unfinished) crm_bmp_prof tool (though it is meant for quite a
different purpose, but this kind of 'off-line' analysis can be built
in there quite nicely as it keeps a lot of hacking outside crm114
itself)

For the latter, you will have a little bit easier job, because for
starters that would mean tracking down the 'feature hashes' in the CSS
files with the highest learn counts, which still requires stuff like
my crm_analysis_mark() code to get access to the original words.


Then, before you consider going for it, here's a slew of caveats:

Hyperspace et al use VT (Vector Tokenization) which mixed word hashes
into final feature hashes. Mixing is good for classification, and
makes your life a lot harder, because it means feature hashes are
basically SERIES of words and I don't keep track of that (yet) in the
GerH experimentals.

OSB and friends may look harmless, but they have the same mixing thing
embedded in the main classifier code, producing the same harm. In
other words: using something like the set of

    MARK_HASH_VALUE,            // register hash values
    MARK_HASH64_VALUE,          // register hash values
    MARK_HASH_CONTINUATION,     // continuation of the preceding
MARK_HASH[64]_VALUE: stores remainder of the hashed string
    MARK_VT_HASH_VALUE,         // register VT-based hash values

marker calls from GerH experimental will give you enough output to
transfrom *word* hashes back to the original words, but there
currently is nothing in there yet to transform 'feature hashes' back
to the 'word hash' series, which means you've got extra work to do in
the 'logging' dept. there, even if you take the latest GerH as
starting point.

All classifiers perform this process, either through the VT code or by
themselves, to arrive at feature hashes:
text words (regex matches) --> word hashes --mixing--> feature hash
Watch plural/singular words in process pipe above. Besides, one word
hash will end up in multiple feature hashes.
That's two transformations, which you /can/ roll into one, but that
will blow up your lookup set even more as one feature hash really is a
whole text /phrase/ AND one text word alone already produces multiple
hash features (phrasings) as it is mixed with various context
elements: one average, say four feature hashes per text word and a
text phrase for a feature hash is two to 4 or 5 words, off the cuff
ballpark figures, but it'll give you an idea of scale. I have
collected all unique 'words' in about 4 years of emails of mine (4
years ham, 1 year spam) and it's a 5.8 million 'words', thanks to all
the gibberish in spam. Multiply that by 4 to get a ballpark figure for
the number of feature hashes...

Which glosses over the tracing I/O itself, which basically is a
compact binary running commentary on your classifier activity. It's
quite fast (and a little overhaul in the buffering I/O code would make
it even faster) but it'll slow you down anyway.

Of course, you can do it another way and off-line the whole thing by
taking out the regex word extractor, hasher and VT/hash mixer to
reproduce all feature hashes off-line and link them up with their
originating data (collection of words which produced the feature hash)
but that would be trading on-line speed for double effort as you'll be
doing the same again off-line.


This is only to get the data for reverse transformation from feature
hash to text words.


Next arena of brain crunch is the 'significance' section, as you'll
need to analyze that particular classifier's scoring function and
determine how significantly the feature contributed to the overall
score. You can simply stick to 'highest learncount' is the highest
weight, so we'll go with that, or take it further. I'd say: keep it it
as simple as you can, because it's already a lot of work anyway.


Forget the crm114 -t -T stuff Bill mentioned, that's only going to
work if you like to OD on incomprehensible logfile overload. (And it
doesn't contain the all transformation stuff listed above anyhow.)



So... you still want to do it? ;-)



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/