JR's patch to not send confirms for obvious spam

I am the honored recipient of an average of 350 spam messages a day
which ASK has been filtering out for me very successfully for quite some
time now.

In recent weeks and months, however, I have noticed that the number of
spam messages getting through has been increasing.  On some days as much
as 10-15% was getting through, which at 350/day becomes a noticably
annoying amount.

On analysis it seems that all of the spam that is getting through falls
into two categories:
	- message appears to be from mailer-daemon
	- spammer is replying to ASK's confirm message

I would like to offer the patch below which addresses both of these
problems.  Using it, I am back to zero spam, and so far, no false
positives.  This is a patch to ASK 2.4.1-1.

The patch makes four changes:

1.  The mailer-daemon problem is due spam matching ASK's mailer-daemon
    check when in fact the message is not a mailer-daemon or postmaster
    message.  Much spam contains headers something like:
    	From: <[email protected]>
    	Reply-To: <postmaster@##.##.##.##>
    (##.## = some IP address).  ASK 2.4.1-1 uses the Reply-To header in
    preference to the From header when checking to see if the message
    is from mailer-daemon.  This is wrong.  The Reply-To header should
    not be checked in this case.  The patch fixes this.
    
2.  The second problem, that of spammers replying to confirm messages,
    can obviously be reduced if the number of confirms sent out in the
    first place is reduced.  On looking at the spam I get, I observed
    that in over 50% of the spam to which ASK is sending confirms,
    none of my recipient addresses were included in the message headers.

    This can happen in three cases, (a) if a message is sent to an alias
    that forwards to me, (b) if the message is to a mailinglist I am
    on, or (c) if the message is spam and the spammer used my address
    in the envelope but a different recipient list (or no list) in the
    headers.  I can match my aliases and mailinglists in my whitelist.
    The remaining messages can simply be classed as Junk and not sent
    a confirm.
    
    By doing this, the number of confirms ASK sends is reduced by
    over 50%.  One would presume that this would therefore reduce the
    chances of spammers replying to confirms by about half, and thereby
    cut spam that is getting through by about half.  In fact, however,
    I have seen no spam in my inbox at all since developing this patch,
    suggesting that the spammers that do reply to confirms tend to be
    predominantly those who don't include my address in their message.
    Perhaps something to do with their use of automated scripts to
    send bulk, non-personalized email and to look for and reply to
    ASK's confirms.
    
    So rather than a 50% decline in spam getting through, this change
    seems to give a considerably better result.
    
    Before using this patch, it is necessary to make sure that all
    your genuine incoming mail is to an address on your rc_mymails
    list, or it will match your whitelist, or it contains your mailkey.
    Since some users may not wish work in this way, I have introduced a
    new .askrc variable to entirely enable or disable this functionality.
    By default, ASK behaves as before, but if you include:
    
        # Set if you don't want to send ASK confirms for mail which
        # doesn't include any of your rc_mymails addresses
        rc_dont_confirm_unknown_recip = true
    
    in your .askrc, you get the new functionality.

3.  The patch also restores the use of the rc_junkmailbox and
    rc_bulkmailbox variables.  If these variables are defined, email
    classified as Junk or Bulk will be saved in those files.  If the
    variables are not defined, such email is saved in the queue dir,
    which is the current functionality.

4.  Finally, the patch also adds a blank line in the log at the start
    of each ASK run and corrects an instance where "mailer-daemon" was
    logged as "mailer_daemon".  This is just to make the log a little
    easier to read and search.

OK, here's the patch.  I welcome folks' comments.  These diffs are all
to files in /usr/lib/ask.  To apply the patch, save it in a file, and
do something like this:
	# cd /usr/lib
	# mv ask ask-2.4.1-1
	# mkdir ask-2.4.1-1-jr
	# ln -s ask-2.4.1-1-jr ask
	# cp ask-2.4.1-1/* ask
	# cd ask
	# patch -p1 <patchfile
Then edit your .askrc and add the rc_dont_confirm_unknown_recip variable
as shown above.

Good luck.





diff -c ask-2.4.1-1/askconfig.py ask-2.4.1-1-jr/askconfig.py
*** ask-2.4.1-1/askconfig.py	2003-05-24 11:27:42.000000000 -0400
--- ask-2.4.1-1-jr/askconfig.py	2004-05-04 19:47:32.000000000 -0400
***************
*** 134,139 ****
--- 134,141 ----
  		self.rc_confirm_mailbody   = self.__expand("rc_confirm_mailbody",   mandatory = 0)
  		self.rc_blacklist_mailbody = self.__expand("rc_blacklist_mailbody", mandatory = 0)
  
+ 		self.rc_dont_confirm_unknown_recip = self.__expand("rc_dont_confirm_unknown_recip", mandatory = 0, boolean = 1)
+ 
  		self.rc_confirm_dirs    = self.__expand("rc_confirm_dirs",   listmode = 1, mandatory = 0)
  		self.rc_blacklist_dirs  = self.__expand("rc_blacklist_dirs", listmode = 1, mandatory = 0)
  		self.rc_confirm_langs   = self.__expand("rc_confirm_langs",  listmode = 1, mandatory = 0)
diff -c ask-2.4.1-1/askmain.py ask-2.4.1-1-jr/askmain.py
*** ask-2.4.1-1/askmain.py	2003-05-24 11:27:42.000000000 -0400
--- ask-2.4.1-1-jr/askmain.py	2004-05-06 09:56:17.000000000 -0400
***************
*** 81,86 ****
--- 81,87 ----
  		"""
  		
  		## Initial Logging
+ 		self.log.write(1, "")
  		self.log.write(1, "----- ASK v%s Started -----" % askversion.AskVersion.version)
  
  		self.msg.read(filehandle)
***************
*** 133,139 ****
  		## an invalid sender address)
  
  		if self.msg.is_from_mailerdaemon():
! 			self.log.write(1, "Sender is mailer_daemon")
  
  			if self.msg.sent_by_ask():
  				self.log.write(1, "Header X-AskVersion found. Probable invalid reply from a confirmation message. Ignoring.")
--- 134,140 ----
  		## an invalid sender address)
  
  		if self.msg.is_from_mailerdaemon():
! 			self.log.write(1, "Sender is mailer-daemon")
  
  			if self.msg.sent_by_ask():
  				self.log.write(1, "Header X-AskVersion found. Probable invalid reply from a confirmation message. Ignoring.")
***************
*** 228,236 ****
  		if self.msg.is_queued():
  			self.log.write(1, "Message is already queued. Won't resend.")
  		else:
! 			self.log.write(1, "Sending confirmation.")
! 			self.msg.queue_mail()
! 			self.msg.send_confirmation()
  
  		return(self.config.RET_PROCMAIL_STOP)
  
--- 229,240 ----
  		if self.msg.is_queued():
  			self.log.write(1, "Message is already queued. Won't resend.")
  		else:
! 			if self.msg.match_recipient() == "" and self.config.rc_dont_confirm_unknown_recip:
! 				self.msg.junk_mail("(Junk) Message with none of our addresses in recipient list")
! 			else:
! 				self.log.write(1, "Sending confirmation.")
! 				self.msg.queue_mail()
! 				self.msg.send_confirmation()
  
  		return(self.config.RET_PROCMAIL_STOP)
  
diff -c ask-2.4.1-1/askmessage.py ask-2.4.1-1-jr/askmessage.py
*** ask-2.4.1-1/askmessage.py	2003-05-24 11:27:42.000000000 -0400
--- ask-2.4.1-1-jr/askmessage.py	2004-05-06 08:58:47.000000000 -0400
***************
*** 428,434 ****
  		Returns true if this message comes from MAILER-DAEMON. False otherwise
  		"""
  
! 		if re.match("mailer-daemon|postmaster@|<>", self.get_sender()[1], re.IGNORECASE):
  			return 1
  		else:
  			return 0
--- 428,434 ----
  		Returns true if this message comes from MAILER-DAEMON. False otherwise
  		"""
  
! 		if re.match("mailer-daemon|postmaster@|<>", self.get_sender(ignore_header = "Reply-To")[1], re.IGNORECASE):
  			return 1
  		else:
  			return 0
***************
*** 464,470 ****
  		"""
  
  		self.log.write(1, "  junk_mail(): Queueing Junk Message")
! 		self.queue_mail(custom_header = custom_header)
  		return 0
  
  	#------------------------------------------------------------------------------
--- 464,474 ----
  		"""
  
  		self.log.write(1, "  junk_mail(): Queueing Junk Message")
! 		if self.config.rc_junkmailbox:
! 			self.mail.deliver_mail(self.config.rc_junkmailbox, self.tmpfile, custom_header)
! 		else:
! 			self.queue_mail(custom_header = custom_header)
! 
  		return 0
  
  	#------------------------------------------------------------------------------
***************
*** 475,481 ****
  		"""
  
  		self.log.write(1, "  bulk_mail(): Queueing Bulk Message")
! 		self.queue_mail(custom_header = custom_header)
  		return 0
  
  	#------------------------------------------------------------------------------
--- 479,489 ----
  		"""
  
  		self.log.write(1, "  bulk_mail(): Queueing Bulk Message")
! 		if self.config.rc_bulkmailbox:
! 			self.mail.deliver_mail(self.config.rc_bulkmailbox, self.tmpfile, custom_header)
! 		else:
! 			self.queue_mail(custom_header = custom_header)
! 
  		return 0
  
  	#------------------------------------------------------------------------------
***************
*** 513,521 ****
  
  		queueFileName = self.queue_file()
  
! 		self.log.write(1, "  send_confirmation(): Sending confirmation to %s" % self.get_real_sender()[1])
  
! 		self.mail.mailfrom = self.__match_recipient()
  
  		## Add Precedence: bulk and (if appropriate) "In-Reply-To"
  
--- 521,532 ----
  
  		queueFileName = self.queue_file()
  
! 		self.mail.mailfrom = self.match_recipient()
  
! 		if self.mail.mailfrom == "":
! 			self.mail.mailfrom = self.config.rc_mymails[0]
! 
! 		self.log.write(1, "  send_confirmation(): Sending confirmation from %s to %s" % (self.mail.mailfrom, self.get_real_sender()[1]))
  
  		## Add Precedence: bulk and (if appropriate) "In-Reply-To"
  
***************
*** 561,566 ****
--- 572,579 ----
  
  			os.unlink(tempFile)
  
+ 		return 1
+ 
  	#------------------------------------------------------------------------------
  
  	def	__check_confirm_list(self, address):
***************
*** 744,750 ****
  
  		self.log.write(1, "  send_nastygram(): Sending nastygram back")
  
! 		self.mail.mailfrom = self.__match_recipient()
  
  		self.mail.send_mail(self.get_real_sender()[1],
  			"Please stop sending me emails", 
--- 757,763 ----
  
  		self.log.write(1, "  send_nastygram(): Sending nastygram back")
  
! 		self.mail.mailfrom = self.match_recipient()
  
  		self.mail.send_mail(self.get_real_sender()[1],
  			"Please stop sending me emails", 
***************
*** 1059,1065 ****
  		fh_output.close()
  
  	#------------------------------------------------------------------------------
! 	def __match_recipient(self):
  		"""
  		This function will do its best to determine which of our rc_mymails was
  		the recipient of the current message. If no entry can be found in the
--- 1072,1078 ----
  		fh_output.close()
  
  	#------------------------------------------------------------------------------
! 	def match_recipient(self):
  		"""
  		This function will do its best to determine which of our rc_mymails was
  		the recipient of the current message. If no entry can be found in the
***************
*** 1070,1081 ****
  		for (recipient_name, recipient_mail) in self.get_recipients():
  			for our_address in self.config.rc_mymails:
  				if recipient_mail == our_address:
! 					self.log.write(1, "  __match_recipient(): Found a match with %s" % our_address)
  					return our_address
  
! 		self.log.write(1, "  __match_recipient(): No Match found. Using %s" % self.config.rc_mymails[0])
  
! 		return self.config.rc_mymails[0]
  		
  #------------------------------------------------------------------------------
  
--- 1083,1094 ----
  		for (recipient_name, recipient_mail) in self.get_recipients():
  			for our_address in self.config.rc_mymails:
  				if recipient_mail == our_address:
! 					self.log.write(1, "  match_recipient(): Found a match with %s" % our_address)
  					return our_address
  
! 		self.log.write(1, "  match_recipient(): No Match found.")
  
! 		return ""
  		
  #------------------------------------------------------------------------------
  


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click