1.4c and inline image spams

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I have just posted spamprobe version 1.4c to sourceforge.  This release adds
a new option, "-o whitelist" for use in combating inline image spams.  More
on that below.  This release also adds a "-o ignore-body" option to ignore
tokens from the message body, fixes a few minor bugs, and updates the man
page.

The primary motivation for this release is to make available the changes
I've made to allow me to completely eliminate inline image spams.  Since
implementing these changes and a set of procmailrc rules neither I nor the
users on my system have received any inline image spams.  All of them have
been caught by procmail/SP with no false positives.

Rather than mess around with parsing images or using OCR I decided to simply
treat all seemingly spammy email with inline images in a special way.  (Note
that SP normally catches most inline image spams, only a few make it past
the filter SP even without these changes.)  I didn't want to completely ban
them (sorry Darxus) because some of my users normally receive legitimate
emails containing them.  Instead I took the approach of blocking them unless
a white list says to keep them.  At first I manually maintained a white list
but that was way too primitive so I decided to let SP implement the white
list for me.

A note on terminology.  Normally a white list is a list of correspondents
whose email you would always accept.  That's not really what I'm doing.
Rather I am using SP in a special variation of "headers-only" mode to
determine if the headers in the email are suspicious or not.  Generally any
email from a normal correspondent will have incredibly hammy headers so
using headers only for scoring is kind of a "virtual white list."

The implementation I chose is controlled by a new command line option:

  -o whitelist

This option causes SP to do several special things:

  * ignores the message body completely
  * ignores new/unique terms
  * scores in "-X" mode (minimum 5 terms but keep scoring
    with terms as long as they are highly spammy or hammy
    with no limit on number of terms used)

These options have the effect of determining whether or not the message is
from a known good or spam source without being confused by terms in the
message body (the random text used to slip past filters).

To catch inline image spams I did the following:

 1. Created a new folder named "inlines" that will receive suspicious
    emails containing inline images.  Think of this as a quarantine
    folder.  I didn't have to use a special folder but I chose to do
    so to make it easier to look for false positives.

 2. Modified my procmailrc file to rescore (using -o whitelist) any
    apparently good email containing inline images.  Messages that pass
    this second test go into my inbox.  Any that fail go into the
    inlines folder.

 3. Modified my cron script that trains on folders periodically to
    perform a train-spam on the inlines folder in the same way it
    does the spamprobe folder (no special options).

Here is the relevant portion of my procmailrc file with the new rule for
quarantining inline image spams:

:0
SCORE=| /usr/local/bin/spamprobe -T train

:0 wf
| formail -I "X-SpamProbe: $SCORE"

:0 :
* ^X-SpamProbe: SPAM
$MAILDIR/.spamprobe/

:0B
* src *= *(3D)? *"? *cid:
{
  :0c
  | /usr/local/bin/spamprobe remove

  :0
  XSCORE=| /usr/local/bin/spamprobe -T -o whitelist score

  :0 wf
  | formail -I "X-SpamProbe-Headers: $XSCORE"

  :0 :
  * ^X-SpamProbe-Headers: SPAM
  $MAILDIR/.inlines/
}

With these changes in place I no longer receive inline image spams.
Hopefully this technique will be useful for others as well.  Note that you
need 1.4c to use the whitelist option.

All the best,
++Brian



-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.5.2 (Build 4075)
Charset: US-ASCII

wsBVAwUBRZ12QDxRyEoJfXIFAQjBkwf+POo+T4qX0ClGmZHXSzuyBn3JxijF52jV
Z/GaB28GBd7t/N6/SQ3vZGDeuebovbQQVyGFOgZLDGbqMxr1Btnv5cZv9tNF7HSo
028gkvflqGyjS790tZFhPyhpt56SFgR4tH0NJqW8GVZsn2OutviGvDn0sfrqEiMm
pfQ3iaguAAKp0mJ3gXFHGIDXvz0GShZGPh+F2ibKO7NmIH0igoM+bfL6B4CDuCjF
keG60v4UxX/+lneSHzJtyyrJ/s8syhVvFDuaFcwkR69TzXW2Uxehuolo7uQ3gri7
LHZHGi8fGckMVcQ01bEPNGuuxGs/IwXoaHqmJ8bWoBhqXnh+5vJKMw==
=Y3xr
-----END PGP SIGNATURE-----

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV