Re: Empty message apparently not tokenized

Date: Mon, 13 Feb 2006 09:51:08 +0100
From: tBB <tbb-MQwlIdhhGh0tX/[email protected]>
To: John Chandler <jpc-WwYSd80vZ+3bI6/[email protected]>
Subject: Re: [Spamprobe-users] Empty message apparently not tokenized

> (I think) I would like spamprobe 
> to tokenize the header and create a decision matrix.
> 
> What do you think?

If SpamProbe is called with the "-H all" option all headers will be 
tokenized.

Regards, Nico


Thanks for the hint,
but I don't think this would be a good way 
to handle the problem, even if it works.

The man page (see below) refers to 
"headers from the email message"
when discussing the -H option,
but it seems clear that this means
what many people would call
"fields in the header of the e-mail message."

There is no reason to think it would be smart
to tokenize all of the fields in _every_ header,
which is what would be required to avoid 
having all messages with no body become false negatives,
assuming that that even does the job.
Presumably Brian Burton avoided tokenizing some header fields
for good reason.

I guess I could try to list all of the header fields
that spamprobe ordinarily tokenizes,
and then try to request through the -H option
that these fields always be tokenized,
but would this actually solve the problem?
I see no reason to think it would.

Appending a blank line to the spam message below
causes the header to be tokenized.
Therefore it seems to me that the failure
to tokenize the header when there is no blank line present
could be considered to be a design problem.

Apparently I could solve the problem by modifying
our copy of spamprobe, or my .procmailrc file,
to append a blank line to every message 
before it is subjected to tokenizing.
If necessary I _will_ do this,
but it sounds to me like a pretty kludgy way 
of solving the problem.

Again I ask, 

   What is a sensible way to avoid
   having all messages with no body
   come through as false negatives?

-----------
John Chandler


+-------------------------------------------+
- Mailto: tbb-MQwlIdhhGh0tX/[email protected]
- No pure HTML mails please
+-------------------------------------------+
Brian Burton:

In the past week or so 
I have received perhaps ten messages 
similar to the one below.
Most of them, although not all,
claim, like this one, to be from optonline.

Running with -T as I always do, I can see that
spamprobe never created a decision matrix or,
apparently, even tokenized the header.
All of these messages have come through 
as false negatives with a spam score of 0.5, 
which I take to be a default value 
for "No tokens found."

(I think) I would like spamprobe 
to tokenize the header and create a decision matrix.

What do you think?


Return-Path: [email protected]
Delivery-Date: Sun Feb 12 17:14:33 2006
Return-Path: <[email protected]>
X-Original-To: [email protected]
Delivered-To: [email protected]
Received: from lh (201-221-207-53.bk11-dsl.surnet.cl [201.221.207.53])
	by a.cs.okstate.edu (Postfix) with SMTP id 9AA4BA0633;
	Sun, 12 Feb 2006 17:13:59 -0600 (CST)
Received: from congruent2 ([127.0.0.1]) by writely.com with Microsoft SMTPSVC(6.0.3790.1830);
	 Sun, 12 Feb 2006 17:15:43 -0600
Message-Id: <7.9.1.5.4.7[14
Date: Sun, 12 Feb 2006 17:13:59 -0600 (CST)
From: [email protected]
To: undisclosed-recipients:;
X-SpamProbe: GOOD 0.5000000 44e527864b85d781d1ba7de482ab0a38
-----
John Chandler



Version 1.4        Last change: December 2005                  10


      -H option

By default SpamProbe only scans a meaningful  subset  of headers
from the email  message  when  searching  for  words  to score.  The -H
option allows the user to specify additional headers  to scan.
Legal values are  "all",  "nox",  "none",  or  "normal".  "all" scans
all headers, "nox" scans all headers except those start- ing with
X-, "none" does not scan headers, and "normal" scans the normal
set of headers.

In addition to those values you can also explicitly  add a header
to the list of headers to process by adding  the  header name in
lower case preceded by a plus  sign.   Multiple  headers can be
specified by using multiple -H options.  For example, to include
only the From and Received headers in your train command you could
run spamprobe as follows:

   spamprobe -Hnone -H+from -H+received train

To process the normal set of headers but  also  add  the SpamAssassin
header X-SpamStatus you could run spamprobe as follows:

   spamprobe -H+x-spam-status train



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642