Re: Empty message apparently not tokenized

> Thanks for the hint,
> but I don't think this would be a good way 
> to handle the problem, even if it works.

Well, it forces SpamProbe to process the headers in this case but 
nevertheless I misunderstood the issue. I was under the impression you 
were suggesting that there are no useful tokens at all in those empty 
mails including the headers SP normally processes. Even if it turned to 
out to be a bug I'd like to comment on it:

Some weeks ago I wrote a mail to Brian about this issue, that mails with 
empty bodys are not caught by SP but to my shame I didn't bother to 
check if SP tokenized those mails at all :( Besides that I wrote a 
filter for my server similar to the one Brian just suggested, I had also 
enabled the mentioned "-H all" option and since then SP correctly 
classified all of those mails so I assumed it was due to the reason 
mentioned above (no useful tokens), therefore the hint.

> There is no reason to think it would be smart
> to tokenize all of the fields in _every_ header,
> which is what would be required to avoid 
> having all messages with no body become false negatives,
> assuming that that even does the job.
> Presumably Brian Burton avoided tokenizing some header fields
> for good reason.

I wouldn't be too sure that mails with empty bodys are caught if this 
bug is fixed because many of them originate from zombies so the tokens 
SP pulls out might be not that useful either. See the related discussion 
on this list a while ago.

As I mentioned above, I have the "-H all" option enabled since a few 
weeks and the processing of all headers noticeably increased the overall 
accuracy of SpamProbe and that *not* only for those mails with empty 
bodys (of which several hundred reach my server each day). That suggests 
that there are rather useful tokens in the headers which are not 
processed by default. Besides, there are quite a few Bayesian 
classifiers out which do process all headers by default so I wouldn't go 
that far to say it is not smart per se.

Nico

-- 
+-------------------------------------------+
- Mailto: tbb-MQwlIdhhGh0tX/[email protected]
- No pure HTML mails please
+-------------------------------------------+


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642