Re: Re: html tag stripping

--3lcZGd9BuhuYXNfi
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Looks good to me!  Until such a beast arrives, I'm trying out a quick perl
preprocessor I wrote using some ideas suggested by various people.  I'm not
actually sure (yet) how much better results I'm getting; I'll probably go back
and run it on my old spam to see how much it improves/degrades accuracy, but
just in case you'd like to give it a try, I've attached it.  Usage is just
basically :

cat message | ifilepp | ifile -this -that -other

Don't bother reporting bugs; I'm still hacking.

--David Bushong

On Sun, Jul 06, 2003 at 10:21:26PM -0400, Jonadab the Unsightly One wrote:
>
> ...
> 
> Perhaps a plugin architecture is in order -- ifile could parse the
> message into sections, each section having a given content-type and
> encoding, and then for each section see if there is a preprocessor
> plugin installed for that encoding (if so use it) and content-type (if
> so, use it) before proceeding.
> 
> By "plugin" here I don't mean necessarily a dynamic library; a call to
> an external program could work if the interface were well-defined.
> Frankly, the interface could be as simple as ifile passing the raw
> data on standard input to the preprocessor and using its standard
> output as the decoded/preprocessed content.  That might be considered
> inefficient, but it would work, and it would establish a low-bar entry
> level for people writing preprocessor plugins, and the performance hit
> would only be taken when the preprocessors were being used,
> presumably.  What preprocessor command (if any) to use for various
> encodings and types of content could just be specified in the ifile
> configuration.
> 
> Am I making any sense?
> 
> 
> 
> _______________________________________________
> Ifile-discuss mailing list
> [email protected]
> http://mail.nongnu.org/mailman/listinfo/ifile-discuss

--3lcZGd9BuhuYXNfi
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=ifilepp

#!/usr/bin/perl -w
#
# ifilepp -- ifile preprocessor: ifilepp -h for usage
#

use strict;
use Getopt::Std;
use vars qw($defmax);

## variable declaration
my (%opt, $bytes, $maxBytes, %validTag, $tagBuf, $lineBuf, $base64Printed);

## configuration
$defmax = 4096;

for (qw(
    html head body table td tr tbody script form input frame p ul ol dl b i a 
    li center title img meta div span select option optgroup
)) 	{ 
    $validTag{uc()}++; 
    $validTag{lc()}++; 
    $validTag{'/' . uc()}++;
    $validTag{'/' . lc()}++;
}

## startup and options
$0 =~ s,.*/,,;
getopts('hHBMNm:', \%opt);
usage() if $opt{h};

## intialization
$maxBytes = $opt{m} || $defmax;
$tagBuf = '';
$lineBuf = '';
$bytes = 0;
$base64Printed = 0;

## main loop
while (<>) {
    ## skip header
    if (1 .. /^\r?$/) {
	print;
	next;
    }

    ## max size
    if (($bytes += length()) > $maxBytes) {
	print($tagBuf, $lineBuf, <>);
	exit;
    }

    ## skip base64 data
    unless ($opt{M}) {
	if (/^Content-Transfer-Encoding:\s+base64/i../^--/) {
	    print "BaseSixtyFour\n" unless $base64Printed++;
	    next;
	}
	$base64Printed = 0;
    }


    ## join ='ed lines
    if (/=\r?$/) {
	$lineBuf .= $`;
	next;
    } elsif (length($lineBuf)) {
	$_ = $lineBuf . $_;
	$lineBuf = '';
    }

    ## Limited Entity Substitution
    s/&nbsp;?/ /gi;

    ## BadHTML
    s/(\S*)<([^\s>]*)>(\S*)/$validTag{$2} ? $& : " $1$3 BadHTML "/ge
	unless $opt{B};

    ## NonEnglish
    s/\b\w*[\x80-\xff]+\w*\b/ NonEnglish /g unless $opt{N};

    ## HTML tags
    unless ($opt{H}) {
	## complete HTML tags
	s/<[^>]+>//g;

	## close HTML tags
	if (/^[^<]+>/ && length($tagBuf)) {
	    $tagBuf = '';
	    $_ = $';
	}

	## open HTML tags
	if (/<[^>]+$/) {
	    if (length($tagBuf)) {
		print $tagBuf;
		$tagBuf = '';
	    }
	    $tagBuf = $&;
	    $_ = $`;
	} elsif (length($tagBuf)) {
	    $tagBuf .= $_;
	    next;
	}
    }

    print;
}

## subroutines
sub usage {
    die <<EOF;
usage: $0 [-H] [-B] [-M] [-m bytes] [file1 [...]]
       -H: don't strip HTML tags
       -B: don't replace "bad" HTML tags with "BadHTML"
       -M: don't replace base64 mime sections with "Base64"
       -N: don't replace non-english words (any 8-bit chars) with "NonEnglish"
       -m: maximum number of bytes to check (default: $defmax)
EOF
}

--3lcZGd9BuhuYXNfi
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

_______________________________________________
Ifile-discuss mailing list
[email protected]
http://mail.nongnu.org/mailman/listinfo/ifile-discuss

--3lcZGd9BuhuYXNfi--