Re: Release DSPAM 3.9.1 ?

Stevan Bajić <[email protected]> Thu, 16 Jun 2011 20:44:24 +0200

Newsgroups	gmane.mail.spam.dspam.devel
Message-ID	<[email protected]>

On Thu, 16 Jun 2011 17:05:21 +0100
Carlo Rodrigues <[email protected]> wrote:

> Hey Stevan!
> 
Hello Carlo,


> On 06/15/2011 08:32 PM, Stevan Bajić wrote:
> > I just extracted the subject line from the mbox file and made this 
> > quick and dirty Perl file for testing:
> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> > #!/usr/bin/perl -w
> > #
> >
> > require "htmlize.pl";
> >
> > my $header = 'Subject: =?UTF-8?B?RWluIHNjaMO2bmVzIFdvY2hlbmVkZSEgTnVyIDIgVGFnZSBSYWJhdHQsIEhhbmR5IC0xMjfigqwsIFRhYmxldCBQQyAtMzDigqw=?=';
> > if ($header =~ /^(.*?)=\?([^?]+)\?([qb])\?([^?]*)\?=(.*)$/is) {
> >    print htmlize_chars($header) . "\n";
> > } else {
> >    print "Not encoded\n";
> > }
> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >
> > I copied the htmlize.pl from GIT into the same directory and run the Perl script and get this as output:
> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> > theia ~ # perl ./csmr.pl
> > Subject: Ein sch&ouml;nes Wochenede! Nur 2 Tage Rabatt, Handy -127&euro;, Tablet PC -30&euro;
> > theia ~ #
> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >
> > IMHO this looks okay. It's German and say "Have a nice weekend! Only 2 days left for discount, mobile handset -127€, tablet PC -30€".
> >
> > So the htmlize_chars in htmlize.pl is IMHO working correctly. Can you check on your install if you get the same result with the above Perl script?
> >
> Yes, the perl script worked just fine with that Subject string.
> The problem with those errors was on the htmlize() fuction on dspam.cgi.
> After playing around with several types of strings, I found that this 
> new htmlize function, in addition to the new htmlize.pl 1.02 you sent 
> June 10th, works flawlessly for all cases:
> 
> sub htmlize {
>    #
>    # Replace some characters
>    # to be HTML characters
>    #
>    my($text) = @_;
> 
>    use Encode;
>    use HTML::Entities;
> 
>    if ($text =~ /[\xC2-\xDF][\x80-\xBF]/) {
>      $text = decode("utf8", $text);
>      $text = encode_entities(decode_entities($text));
>    }
> 
>    if ($text =~ /^(.*?)=\?([^?]+)\?([qb])\?([^?]*)\?=(.*)$/is) {
>      if (-r "htmlize.pl") {
>        require "htmlize.pl";
>        $text = htmlize_chars($text);
>      }
>    }
> 
>    return $text;
> }
> 
> 
> I really appreciate your help. Quarantine UI is looking gorgeous now :) 
> Thank you.
> 
try this attached patch. It's way smaller than the old one and does avoid double decoding/encoding. Should IMHO yeald the same result as your code but be slightly faster when processing. You need to apply the patch against GIT Master.

Does it work as expected? If so tell me than I will merge it into GIT Master.


> Carlo
> 
-- 
Kind Regards from Switzerland,

Stevan Bajić

------------------------------------------------------------------------------
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev

_______________________________________________
Dspam-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspam-devel

2918624.patch (application/octet-stream, 3.4 KB)

diff --git a/webui/cgi-bin/configure.pl.in b/webui/cgi-bin/configure.pl.in
index befbf10..5dc6863 100755
--- a/webui/cgi-bin/configure.pl.in
+++ b/webui/cgi-bin/configure.pl.in
@@ -1,8 +1,8 @@
 #!/usr/bin/perl
 
-# $Id: configure.pl,v 1.05 2010/01/03 14:39:13 sbajic Exp $
+# $Id: configure.pl,v 1.06 2011/06/08 22:22:35 sbajic Exp $
 # DSPAM
-# COPYRIGHT (C) DSPAM PROJECT 2002-2010
+# COPYRIGHT (C) DSPAM PROJECT 2002-2011
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -43,6 +43,7 @@ $CONFIG{'HISTORY_SIZE'}	= 799;		# Number of items in history
 $CONFIG{'HISTORY_PER_PAGE'} = 100;	# Number of items per page
 $CONFIG{'HISTORY_DUPLICATES'} = "yes";	# Wether to show duplicate entries in history "yes" or "no"
 $CONFIG{'HISTORY_HTMLIZE'} = "no";	# Wether to HTML-ize sender and subject in history "yes" or "no"
+$CONFIG{'QUARANTINE_HTMLIZE'} = "no";	# Wether to HTML-ize sender and subject in quarantine "yes" or "no"
 $CONFIG{'MAX_COL_LEN'}	= 50;		# Max chars in list columns
 $CONFIG{'SORT_DEFAULT'}	= "Rating";	# Show quarantine by "Date" or "Rating"
 $CONFIG{'3D_GRAPHS'}	= 1;		# 0=graphs in 2D, 1=graphs in 3D
diff --git a/webui/cgi-bin/dspam.cgi b/webui/cgi-bin/dspam.cgi
index dae53d1..35ae687 100755
--- a/webui/cgi-bin/dspam.cgi
+++ b/webui/cgi-bin/dspam.cgi
@@ -1,8 +1,8 @@
 #!/usr/bin/perl
 
-# $Id: dspam.cgi,v 1.55 2010/08/03 12:52:06 sbajic Exp $
+# $Id: dspam.cgi,v 1.56 2011/06/08 22:20:45 sbajic Exp $
 # DSPAM
-# COPYRIGHT (C) DSPAM PROJECT 2002-2010
+# COPYRIGHT (C) DSPAM PROJECT 2002-2011
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -1227,6 +1227,12 @@ sub DisplayQuarantine {
 
     if ($alert) { $rowclass="rowAlert"; }
 
+    # HTMLize special characters
+    if ($CONFIG{'QUARANTINE_HTMLIZE'} eq "yes") {
+      $new->{'Subject'} = htmlize($new->{'Subject'});
+      $new->{'From'} = htmlize($new->{'From'});
+    }
+
     $new->{'Sub2'} = $new->{'X-DSPAM-Signature'};
     if (length($new->{'Subject'})>$CONFIG{'MAX_COL_LEN'}) {
       $new->{'Subject'} = substr($new->{'Subject'}, 0, $CONFIG{'MAX_COL_LEN'} - 3) . "...";
@@ -1618,14 +1624,26 @@ sub htmlize {
   #
   my($text) = @_;
 
-  use Encode;
-  use HTML::Entities;
+  my $has_encode = eval{require Encode;};
+  my $has_html_entities = eval{require HTML::Entities;};
 
   if ($text =~ /^(.*?)=\?([^?]+)\?([qb])\?([^?]*)\?=(.*)$/is) {
-    $text = encode_entities(decode($2, Encode::decode('MIME-Header', $text)));
-  } else {
-    $text = decode("utf8", $text) if ($text =~ /[\xC2-\xDF][\x80-\xBF]/);
-    $text = encode_entities(decode_entities($text));
+    if ($has_encode) {
+      $text = Encode::decode($2, Encode::decode('MIME-Header', $text));
+    }
+  } elsif ($text =~ /([\xC2-\xDF][\x80-\xBF]
+                     | \xE0[\xA0-\xBF][\x80-\xBF]
+                     |[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}
+                     | \xED[\x80-\x9F][\x80-\xBF]
+                     | \xF0[\x90-\xBF][\x80-\xBF]{2}
+                     |[\xF1-\xF3][\x80-\xBF]{3}
+                     | \xF4[\x80-\x8F][\x80-\xBF]{2})/x) {
+    if ($has_encode) {
+      $text = Encode::decode("utf8", $text);
+    }
+  }
+  if ($has_html_entities) {
+    $text = HTML::Entities::encode_entities(HTML::Entities::decode_entities($text));
   }
   if ($text =~ /[\xC2-\xDF][\x80-\xBF]/) {
     if ((-e "htmlize.pl") && (-r "htmlize.pl")) {