Re: find and replace, and other questions

On Sunday, December 12, 2010, Kurt Bigler <kkb-qsUyPSV3HvqUK90frp/[email protected]> wrote:
> Back after years away. Not much going on here it seems.
>
Last word was that devs would focus on decoupling the classifiers into a
stand-alone library.

I've had some fun with MIME, SMTP, CGI, web APIs and URL-encoding with the
CRM scripting language. My Twitter code broke when Twitter made OAuth
mandatory, but I have a CRM-114 driven CGI site that uses eval on templates
with embedded :*:variables: to serve dynamic content that it obtains from
the CGI environment and a non-SQL server application. I also tried building
a chat bot that pushed information from the server app I'm using, but piping
a synchronous communication protocol through socat wasn't going to support
more than a handful of users.

So CRM is doing fun and useful stuff still, but the newbies have grown.

> I got stuck years ago when I could not manage the transition to mailreaver
> which did not fit within my framework. I didn't have time to rework
> everything, so gave up on doing any more upgrades. Now I can give it a
> little burst of effort.
>
>
> To the point:
>
> I am trying to understand how (if at all possible) to do regex stuff with
an
> LHS and an RHS, i.e. "find and replace" with regex on the LHS and \1 \2 \3
> etc. on the RHS. Is this even supported? Or is \1 only useful to indicate
> repetition of a substring in the LHS?

I'm not sure how LHS and RHS apply in a declensional language. ;)

Find is "match". You can capture substrings in variables. Replace is
"alter".

{
# Discard match. Keep substring.
match (:: :t:) /(a) hacker/
alter (:t:) /the/
}

> Doc on ALTER has this little statement about expanding \-constants but I'm
> not sure what that means, maybe just octal constants, etc.

Right. Constants like \n, not back references.

> I was hoping that MATCH would set some magic variables that ALTER could
> reference via \1 \2 \3 etc. (It certainly seems like match and alter
> *could* be made to do that.)

Not magic. Just normal program variables.

Find and replace are discrete operations. Back references don't propagate
from the match statement to alter so you use real variables. That's much
less confusing than magic.

> Even if I had to do something like
>
> match /blah/ [ :: :_dw0: :_dw1: :_dw2: :_dw3: ]
>
> combined with a convention that let alter use \N to reference these
> variables.
>
> Reason: I want to use \1 \2 \3 in the RHS of rewrite rules. This seems a
> lot cleaner than hacking mailfilter.crm, pushing the rewrite functionality
> there to get more expressiveness. I'm sure I could find some way hack the
> rewrite rule code to accomplish this but it probably requires adding more
> levels of evaluation which might hurt performance.

In more than one way - speed and accuracy.

> In short I wanted to use rewrite rules to accomplish substitutions that
> would facilitate classification, e.g. by parsing URLs differently,
> recognizing punctuation in the middle of words, etc. It seemed like the
> really clean place to put such functionality.
>
> Any clues?

Testing is your friend. Most attempts to preprocess data proposed to this
list have been counter productive. The reason is obvious in retrospect.
Processing that makes variations more obvious to regex based filters
destroys data that may be of value to classifiers.

Match and alter in a loop with liaf would be great for dealing with
arbitrary encoding depth exploits to URL encoding and HTML that spammers use
to get mail clients to display stuff that gets by Bayesian filters, but
that's a result of the fact that Bayes theorem doesn't attempt to treat
proximity as a causal relationship. Some installations try to work around
that limitation by making tokens of pairs of words, which is easily defeated
by inserting noise between obvious spam words. Markov and OSB in CRM as
related tokens in a chain with a length of 5, with the consequence that the
filter learns to recognize the garbage as a spam indicator.

> At the same time it seemes a little tricky to use such substitutions in a
> way that has significant power with OSB, i.e. without at least some
> trigrams. (Maybe that needs some explanation, but for now I'll just throw
> it out.) Makes me wish for a classification method that would let me
> enumerate the N-grams I want to include in training and classification.
> Since they are enumerable, just a list of integers would do the trick,
> together with a handy reference table of the relevant canonical
> permutations.
>
> Does the OSB code really depend on everything being bigrams?

OSB is derived from Markov in much the same way as a Roman arch is derived
from a load bearing wall. There's a computational efficiency achieved by
making the bigrams sparce that preserves the strength of the Markov chain in
much the same way as an arch conserves material while preserving much of the
physical strength of a wall. Trigrams would just reintroduce unnecessary
redundancy because the tripartite relationships are already implicit in the
evaluation of sparce bigrams. This is the reason that OSB has accuracy
characteristics similar to Markov.

> Meanwhile I have some tricks in mind to make the best of bigrams for this,
> but thought I'd ask whether anyone else has come to think bigrams are a
> little bit limiting, even while the more general set of -grams may be
> overkill.

In the absence of new algorithms, training, tokenization, and classifier
choice are the route to improving accuracy.

For my part, I'd love to hear about the status of crmlib. As much as I've
loved using the CRM-114 scripting language, I've had trouble pushing
adoption in my little hobby circle... and I've reached a complexity point in
my application where I'd rather debug a PEG than a regex.

Chris

------------------------------------------------------------------------------
Oracle to DB2 Conversion Guide: Learn learn about native support for PL/SQL,
new data types, scalar functions, improved concurrency, built-in packages, 
OCI, SQL*Plus, data movement tools, best practices and more.
http://p.sf.net/sfu/oracle-sfdev2dev

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general