Re: Re: Empty message apparently not tokenized

<snip>
> Appending a blank line to the spam message below
> causes the header to be tokenized.
> Therefore it seems to me that the failure
> to tokenize the header when there is no blank line present
> could be considered to be a design problem.
>
> Apparently I could solve the problem by modifying
> our copy of spamprobe, or my .procmailrc file,
> to append a blank line to every message
> before it is subjected to tokenizing.
> If necessary I _will_ do this,
> but it sounds to me like a pretty kludgy way
> of solving the problem.
>
> Again I ask,
>
>    What is a sensible way to avoid
>    having all messages with no body
>    come through as false negatives?
>
> -----------
> John Chandler


I can't offer up any help,
but I've been able to at least reproduce your problem.
(I'm using SP1.4a here on our server.)

I wrote a simple message:
~~~~~~~~~~~~~~~~~~~~~~~~~~
To: somebody
From: somebody else
Date: today
~~~~~~~~~~~~~~~~~~~~~~~~~~

Here's the results of trying to tokenize it:
[/home/xxx]$ spamprobe tokenize msg
[/home/xxx]$

That can't be right.
So, I changed the message to this:
~~~~~~~~~~~~~~~~~~~~~~~~~~
To: somebody
From: somebody else
Date: today

my body
~~~~~~~~~~~~~~~~~~~~~~~~~~

Again I tried to tokenize it:
[/home/xxx]$ spamprobe tokenize msg
0.3000000       0       0       1  Hto_someobody
0.3000000       0       0       1  Hfrom_somebody
0.3000000       0       0       1  Hfrom_else
0.3000000       0       0       1  Hfrom_somebody else
0.3000000       0       0       1  my
0.3000000       0       0       1  body
0.3000000       0       0       1  my body
[/home/xxx]$

So obviously there's some tokens in this message.


Just for fun, I tried the message with no body on a copy of SP1.0a I
had laying around and it worked.
Can anybody else reproduce this?


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642