Re: find and replace, and other questions

Thanks for the thoughts.  More below...


On 12/12/10 2:57 PM, "Chris Babcock" <[email protected]> wrote:

>> I was hoping that MATCH would set some magic variables that ALTER could
>> reference via \1 \2 \3 etc. (It certainly seems like match and alter
>> *could* be made to do that.)
> 
> Not magic. Just normal program variables.

By magic I meant something currently undocumented.  I was hoping it was
there already since it would have made my day.  And for some reason I just
couldn't believe there would be back-references but no "RHS" with apologies
for the term.  That's just because I've usually used back-references only
from the RHS.  (But I see why it is there anyway.)

My problem (not a showstopper) is that I hoped to accomplish this piece of
work in the compact and distinct rewrites.mfp file.  A .crm include file or
whatever it is called would serve almost as well although the list of rules
will be far less compact and easy to scan during maintenance.  I'll just get
creative with the indenting.

> Find and replace are discrete operations. Back references don't propagate from
> the match statement to alter so you use real variables. That's much less
> confusing than magic.

There's nothing magic about mainstream regex substitution.  As you say,
crm114 has the functionality covered via variables, but mailfilter provides
a rewrite mechanism that is not allowed to use variables.  So it comes to
mind immediately that the problem was long solved so why not use that
solution (\1 \2 \3) which also allows the convenience of keeping these
customizations in a place that was at least intended for customization
(rewrites.mfp), even if not intended for this *type* of customization.

Getting variables into rewrites.mfp isn't a sufficient argument for adding
"RHS" back-reference functionality to the crm114 language, I suppose.  The
only argument then might be some expectation that familiar regex forms be
supported in a regex language, especially when the existing crm114
mechanisms wouldn't seem to need much straining to achieve such a thing,
even if there'd be some details to talk about.  That's what I was thinking.
I don't expect everybody to get all excited.  ;)

>> Even if I had to do something like
>> 
>> match /blah/ [ :: :_dw0: :_dw1: :_dw2: :_dw3: ]
>> 
>> combined with a convention that let alter use \N to reference these
>> variables.
>> 
>> Reason: I want to use \1 \2 \3 in the RHS of rewrite rules. This seems a
>> lot cleaner than hacking mailfilter.crm, pushing the rewrite functionality
>> there to get more expressiveness. I'm sure I could find some way hack the
>> rewrite rule code to accomplish this but it probably requires adding more
>> levels of evaluation which might hurt performance.
> 
> In more than one way - speed and accuracy.

I see unclassifiable short spams with almost no information in them that the
classifier has any way to get to, even though it is there.  Classification
is not working, so it can only get better.  Preprocessing as I intend is
closely related to tokenization.  This can't be a new idea, so maybe there
is nothing of value in it.  But I'll say more below anyway.

>> In short I wanted to use rewrite rules to accomplish substitutions that
>> would facilitate classification, e.g. by parsing URLs differently,
>> recognizing punctuation in the middle of words, etc. It seemed like the
>> really clean place to put such functionality.
>> 
>> Any clues?
> 
> Testing is your friend.

Indeed, I am about to find out.  Maybe a little slowly.  Because I'll apply
new substitutions midstream in live training, then go back to old css files
if something goes unstable.  But if no problems show up and if after a
couple months I see any fewer unclassifiable short messages in Unsure, then
some other month when there's time I can try it with some fresh training
instead of piled on top of 4 years.  (Then I can do the real science with
training and validation.)

> Most attempts to preprocess data proposed to this list
> have been counter productive.

I guess I heard wsy say something like that years ago.  His point probably
referred more to ad-hoc efforts to capitalize on the human conception of
what a filter should do, or what is "obvious", that this tends to backfire
since it can not anticipate how bayes/markov statistics actually operate in
a real corpus.

Maybe you are referring to discussions long long back?  At least in a quick
scan of the last 3 or so years I missed I didn't see *too* much about
preprocessing.

> The reason is obvious in retrospect. Processing
> that makes variations more obvious to regex based filters destroys data that
> may be of value to classifiers.

At least on the surface that doesn't apply.  I certainly have no thoughts
about making variations more obvious to a regex filter, even if I use a
regex filter to *do* the preprocessing.  The *only* point would be to make
variations visible to the classifier that are currently hidden from it.

The hidden things I'm concerned about relate largely to ineffective
tokenization (e.g. URLs treated as a single word in a message with hardly
enough words in the body to make a bigram).

So it may be that changing tokenization is more to the point, but I see
opportunities to expose information to the classifier that ordinary
tokenization can not.  So in effect I'm changing effective tokenization by a
preprocessing step that takes advantage of the existing tokenizer.  In my
case one reason for doing it that way is that I can fade things in more
gradually that would be abrupt with a tokenization change (or else would be
very hard to express) and test stability in a *ongoing* live training
scenario.  And the other reason is to be able to experiment with additional
tweaks that can't be expressed by a tokenization rule.

So I think I'm not naïve to what you're saying, but as you say testing will
tell.
 
> OSB is derived from Markov in much the same way as a Roman arch is derived
> from a load bearing wall. There's a computational efficiency achieved by
> making the bigrams sparce that preserves the strength of the Markov chain in
> much the same way as an arch conserves material while preserving much of the
> physical strength of a wall. Trigrams would just reintroduce unnecessary
> redundancy because the tripartite relationships are already implicit in the
> evaluation of sparce bigrams. This is the reason that OSB has accuracy
> characteristics similar to Markov.

Yes I can see the computational efficiency aspect, and can imagine the proof
of equivalence.  However I think the timing of responsiveness to in-use
training will be different.  Combining independent bigrams to express
"trigram features", I'm guessing convergence would be slower.  But that's
only intuitive (may be incorrect).

But to say it another way I take the equivalence of makovian to bayesian to
be a statement about the steady state, and not about the dynamics of a live
training scenario.  If you can expound about the dynamics that would be
interesting.  I don't read the literature.  ;)

And for my volume, computational efficiency may not matter.  I'd just rather
spend less time training.  But it sounds like the new perhaps compute-hungry
classifiers that may do better with short messages might be a bit too
experimental for me to get into.  So I just go with my curiosity and have
fun with it!

>> Meanwhile I have some tricks in mind to make the best of bigrams for this,
>> but thought I'd ask whether anyone else has come to think bigrams are a
>> little bit limiting, even while the more general set of -grams may be
>> overkill.
> 
> In the absence of new algorithms, training, tokenization, and classifier
> choice are the route to improving accuracy.

Yes, well your inclusion of tokenization there makes me feel on the right
track, since at least half the preprocessing functionality I have in mind
could is provably equivalent to an alternate tokenization.  (Yet the other
half may be more fun.)
 
> For my part, I'd love to hear about the status of crmlib. As much as I've
> loved using the CRM-114 scripting language, I've had trouble pushing adoption
> in my little hobby circle... and I've reached a complexity point in my
> application where I'd rather debug a PEG than a regex.

You're clearly better with crm114 then I ever got to be.  But using it is
still easier than working around it given the development time-cycles I have
available.  And it does challenge some of my habitual thinking in useful
ways which is always a very good thing, and makes it worth the trouble, so
far.

-Kurt



------------------------------------------------------------------------------
Oracle to DB2 Conversion Guide: Learn learn about native support for PL/SQL,
new data types, scalar functions, improved concurrency, built-in packages, 
OCI, SQL*Plus, data movement tools, best practices and more.
http://p.sf.net/sfu/oracle-sfdev2dev