Re: 1.4c and inline image spams

I've been  thinking about this option.  To make it really useful (to me of 
course !)
to have a variant of this argument "-o whiteliest" which does the 
determination of if there are inline images for you.
Thus avoiding having to use .procmail script ... and avoiding yet a Nth pass 
on the email before spamprobe gets it

That is say "-o inline-whitelist" or whatever non-controversial argument 
name (I don't care) that does


IF  body contains inline images
    work as if "-o whitelist" was supplied
ELSE
    ignore "-o whitelist" and score as usual


I would probably ALWAYS specify this argument.    It seems to me that it may 
be very useful that if body contained inline images then its best to score 
only by the headers ... always.

I'm sure I'm missing something obvious ... but it seems right to me ...


-David
-----------------------------------------------------------
David A. Lee
[email protected]
http://www.calldei.com




----- Original Message ----- 
From: "Brian Burton" <[email protected]>
To: "Spamprobe Users" <spamprobe-users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org>
Sent: Thursday, January 04, 2007 3:48 PM
Subject: [Spamprobe-users] 1.4c and inline image spams


> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> I have just posted spamprobe version 1.4c to sourceforge.  This release 
> adds
> a new option, "-o whitelist" for use in combating inline image spams. 
> More
> on that below.  This release also adds a "-o ignore-body" option to ignore
> tokens from the message body, fixes a few minor bugs, and updates the man
> page.
>
> The primary motivation for this release is to make available the changes
> I've made to allow me to completely eliminate inline image spams.  Since
> implementing these changes and a set of procmailrc rules neither I nor the
> users on my system have received any inline image spams.  All of them have
> been caught by procmail/SP with no false positives.
>
> Rather than mess around with parsing images or using OCR I decided to 
> simply
> treat all seemingly spammy email with inline images in a special way. 
> (Note
> that SP normally catches most inline image spams, only a few make it past
> the filter SP even without these changes.)  I didn't want to completely 
> ban
> them (sorry Darxus) because some of my users normally receive legitimate
> emails containing them.  Instead I took the approach of blocking them 
> unless
> a white list says to keep them.  At first I manually maintained a white 
> list
> but that was way too primitive so I decided to let SP implement the white
> list for me.
>
> A note on terminology.  Normally a white list is a list of correspondents
> whose email you would always accept.  That's not really what I'm doing.
> Rather I am using SP in a special variation of "headers-only" mode to
> determine if the headers in the email are suspicious or not.  Generally 
> any
> email from a normal correspondent will have incredibly hammy headers so
> using headers only for scoring is kind of a "virtual white list."
>
> The implementation I chose is controlled by a new command line option:
>
>  -o whitelist
>
> This option causes SP to do several special things:
>
>  * ignores the message body completely
>  * ignores new/unique terms
>  * scores in "-X" mode (minimum 5 terms but keep scoring
>    with terms as long as they are highly spammy or hammy
>    with no limit on number of terms used)
>
> These options have the effect of determining whether or not the message is
> from a known good or spam source without being confused by terms in the
> message body (the random text used to slip past filters).
>
> To catch inline image spams I did the following:
>
> 1. Created a new folder named "inlines" that will receive suspicious
>    emails containing inline images.  Think of this as a quarantine
>    folder.  I didn't have to use a special folder but I chose to do
>    so to make it easier to look for false positives.
>
> 2. Modified my procmailrc file to rescore (using -o whitelist) any
>    apparently good email containing inline images.  Messages that pass
>    this second test go into my inbox.  Any that fail go into the
>    inlines folder.
>
> 3. Modified my cron script that trains on folders periodically to
>    perform a train-spam on the inlines folder in the same way it
>    does the spamprobe folder (no special options).
>
> Here is the relevant portion of my procmailrc file with the new rule for
> quarantining inline image spams:
>
> :0
> SCORE=| /usr/local/bin/spamprobe -T train
>
> :0 wf
> | formail -I "X-SpamProbe: $SCORE"
>
> :0 :
> * ^X-SpamProbe: SPAM
> $MAILDIR/.spamprobe/
>
> :0B
> * src *= *(3D)? *"? *cid:
> {
>  :0c
>  | /usr/local/bin/spamprobe remove
>
>  :0
>  XSCORE=| /usr/local/bin/spamprobe -T -o whitelist score
>
>  :0 wf
>  | formail -I "X-SpamProbe-Headers: $XSCORE"
>
>  :0 :
>  * ^X-SpamProbe-Headers: SPAM
>  $MAILDIR/.inlines/
> }
>
> With these changes in place I no longer receive inline image spams.
> Hopefully this technique will be useful for others as well.  Note that you
> need 1.4c to use the whitelist option.
>
> All the best,
> ++Brian
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: PGP Desktop 9.5.2 (Build 4075)
> Charset: US-ASCII
>
> wsBVAwUBRZ12QDxRyEoJfXIFAQjBkwf+POo+T4qX0ClGmZHXSzuyBn3JxijF52jV
> Z/GaB28GBd7t/N6/SQ3vZGDeuebovbQQVyGFOgZLDGbqMxr1Btnv5cZv9tNF7HSo
> 028gkvflqGyjS790tZFhPyhpt56SFgR4tH0NJqW8GVZsn2OutviGvDn0sfrqEiMm
> pfQ3iaguAAKp0mJ3gXFHGIDXvz0GShZGPh+F2ibKO7NmIH0igoM+bfL6B4CDuCjF
> keG60v4UxX/+lneSHzJtyyrJ/s8syhVvFDuaFcwkR69TzXW2Uxehuolo7uQ3gri7
> LHZHGi8fGckMVcQ01bEPNGuuxGs/IwXoaHqmJ8bWoBhqXnh+5vJKMw==
> =Y3xr
> -----END PGP SIGNATURE-----
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share 
> your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Spamprobe-users mailing list
> Spamprobe-users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/spamprobe-users
> 


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV