Author: fkento
Date: Tue Aug 25 21:16:50 2026
New Revision: 1937456
Log:
Move HTML rendering from Message::Node::rendered() into Handler::HTML
Rendering an HTML part is the HTML handler's job, but handle_html() only
called $node->rendered() to warm a cache and then read the results back.
The parse now happens in the handler; rendered() renders text/plain and
otherwise returns whatever the part's handler published with
set_rendered().
Mail::SpamAssassin::HTML is untouched.
Note this makes the HTML handler load-bearing: with it not loaded, no HTML
is parsed at all -- no rendered body text, no html_results for the html_*
eval rules, and no URIs from href/src/action. Previously only the URI
harvest was lost. v403.pre loads it by default (under the
feature_handlers guard), so an install whose lib/ and rules/ come from the
same version is unaffected; a site running this code against an older
v403.pre, or one that has commented the loadhandler out, is not. The
existing warn in Message::apply_handlers() for a text/html part with no
registered handler covers this; it now says what is actually lost, and has
moved inside the part-walking loop, which reuses the per-node handler
lookup already done there instead of a separate find_parts() scan. Two
cases it used to miss now warn: a message whose only text/html part is a
synthetic child produced by another handler (ICS, Archive), and a config
with no handlers registered at all -- the latter returned early, before
the check.
Node.pm:
- new decode_and_normalize(): decode() plus the charset decision, factored
out of the two branches of rendered() that each had their own copy.
Returns ($text, $character_semantics) so a caller feeding a parser that
must be told whether it is getting bytes or characters can do so.
- set_rendered() takes optional $invisible and $visible arguments, for a
renderer that can tell hidden text from shown text. Both default to the
old behaviour, so existing callers are unaffected.
- rendered() no longer parses HTML; its type gate is text/plain only.
Handler/SVG.pm now publishes the text it extracts with set_rendered(),
under the part's own image/svg+xml type. Previously an SVG part was
rendered only as a side effect of rendered()'s HTML branch, which stamped
it text/html -- so an SVG arriving as an attachment (the common case) was
then dropped from the body by the text/* attachment skip in
get_body_text_array_common().
This is a behaviour change, not just code motion: SVG text now reaches body
rules where it did not before, the same way OCR'd image and PDF text
already do. On t/data/nice/handler_svg the body gains \"SVGSENTINEL Please
view documents on docusign secure link\".
Bayes asks for the MIME-part handlers explicitly on the learn path
(get_body_from_msg). It already got them, but only incidentally: it calls
Message::extract_message_metadata, which does not run handlers, and then
_get_msgdata_from_permsgstatus -> get_uri_list -> get_uri_detail_list,
which does. The handler-extracted text Bayes learns therefore hung on a
call made for its URIs. apply_handlers() is idempotent, so this is a
no-op at runtime; it just stops the token stream depending on that.
One smaller behaviour change: text/plain parts now emit the
\"normalize_charset is off\" debug line, which only the HTML branch used to
log. Debug output only.
Also:
- Pyzor::Digest::Pieces read {rendered} directly, which only worked if
something else had already rendered the part; call rendered().
- Message::get_body_text_array_common no longer pushes an undef into
{metadata}{html_all} when a handler publishes text/html without
html_results.
- t/scan_text_attach.t builds a PerMsgStatus and calls apply_handlers(),
as a real scan does; it previously relied on rendered() lazily parsing
HTML with no handler having run.
Verified behaviour-neutral for HTML: rendered/visible/invisible text,
utf8 flags and the html_results stats are byte-identical across 70
rendered parts from 23 fixtures, with normalize_charset both on and off.
Add note to UPGRADE file
Submitted by: Kent Oyer <[email protected]>
Github: closes #39
Modified:
spamassassin/trunk/UPGRADE
spamassassin/trunk/lib/Mail/SpamAssassin/Handler/HTML.pm
spamassassin/trunk/lib/Mail/SpamAssassin/Handler/SVG.pm
spamassassin/trunk/lib/Mail/SpamAssassin/Message.pm
spamassassin/trunk/lib/Mail/SpamAssassin/Message/Node.pm
spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Bayes.pm
spamassassin/trunk/lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm
spamassassin/trunk/t/scan_text_attach.t
Modified: spamassassin/trunk/UPGRADE
==============================================================================
--- spamassassin/trunk/UPGRADE Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/UPGRADE Tue Aug 25 21:16:50 2026 (r1937456)
@@ -101,6 +101,21 @@ Note for Users Upgrading to SpamAssassin
handlers. Body script is inert and only surfaced when the HTML
part is itself an attachment.
+ IMPORTANT: this handler is now the only thing that renders HTML.
+ If it is not loaded, text/html parts are not parsed at all: body
+ rules and Bayes see no text from them, the html_* eval rules have
+ no results to test, and URIs in HTML links (href, src, action,
+ etc.) are never extracted -- so a mostly-HTML mail stream will
+ score far lower than it did before and spam will be missed. It is
+ loaded by default from v403.pre, so a stock install is unaffected.
+ Check for it if you upgrade over a v403.pre you have edited, or
+ keep an older copy of the file:
+
+ loadhandler Mail::SpamAssassin::Handler::HTML
+
+ SpamAssassin warns once per process when a message contains a
+ text/html part and no handler is registered for it.
+
* JavaScript (text/javascript) collects script text for the new
"script" rule type (like rawbody rules) and adds navigation/
redirect URLs (e.g. window.location = '...') to the URI detail
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Handler/HTML.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Handler/HTML.pm Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Handler/HTML.pm Tue Aug 25 21:16:50 2026 (r1937456)
@@ -26,8 +26,9 @@ Mail::SpamAssassin::Handler::HTML - a MI
=head1 DESCRIPTION
This handler registers as the MIME-part handler for C<text/html>. It
-renders each HTML part by calling C<rendered()> on it -- which parses the HTML
-and caches the rendered text and C<html_results>.
+parses and renders each HTML part, publishing the rendered text back onto the
+part with C<set_rendered()> and caching the parser's findings in
+C<html_results>.
=head1 RETURNS
@@ -61,6 +62,7 @@ use warnings;
use re 'taint';
use Mail::SpamAssassin::Handler;
+use Mail::SpamAssassin::HTML;
use Mail::SpamAssassin::Logger;
our @ISA = qw(Mail::SpamAssassin::Handler);
@@ -82,12 +84,10 @@ sub new {
sub handle_html {
my ($self, $node, $pms) = @_;
- # The ultimate goal is to do the HTML rendering here in the handler, so that
- # Mail::SpamAssassin::Message::Node::rendered() becomes a plain accessor for
- # the cached result. For now we just reuse the rendering code already in
- # rendered(): it parses the HTML and caches the result (and html_results), so
- # later lazy callers of rendered() get a cache hit. Idempotent.
- $node->rendered();
+ # Parse and render the HTML. Idempotent: a part can reach the handler twice
+ # (identical content re-dispatched as a synthetic child), and re-parsing would
+ # just rebuild the same results, so skip straight to the harvest below.
+ $self->_render($node) if !$node->{html_results};
my $results = $node->{html_results} or return [];
@@ -142,4 +142,74 @@ sub handle_html {
return \@parts;
}
+# Parse one text/html part and publish what the parser produced onto the node:
+# the rendered text (all of it, plus the visible-only and invisible-only
+# streams) via set_rendered(), and the parser's findings in {html_results} for
+# the HTML eval rules. Body rules, Bayes and the html_* rules all read these
+# back later; nothing here touches $pms.
+sub _render {
+ my ($self, $node) = @_;
+
+ # Length of the part's decoded bytes, before any charset transcoding -- the
+ # denominator of the html_length/text-length ratio below.
+ my $text_len = length($node->decode // '');
+
+ # An empty part has nothing to parse, but still renders as empty text rather
+ # than as "not rendered at all" -- callers counting text vs. HTML parts (e.g.
+ # BodyEval::check_for_mime_html) have always seen it, and skipping it here
+ # would silently drop it from those counts.
+ if (!$text_len) {
+ $node->set_rendered('', 'text/html');
+ return;
+ }
+
+ # Feed the parser characters where the charset lets us decode them, and tell
+ # it which it is getting: HTML::Parser handles either, but its utf8_mode (and
+ # our own NBSP and word-boundary handling) has to match the actual input.
+ my ($text, $character_semantics) = $node->decode_and_normalize();
+ return if !defined $text || $text eq '';
+
+ # the 1 requires decoded HTML results to be in characters (utf8 flag on)
+ my $html = Mail::SpamAssassin::HTML->new($character_semantics, 1); # object
+
+ $html->parse($text); # parse+render text
+
+ # resulting HTML-decoded text is in perl characters (utf8 flag on)
+ my $rendered = $html->get_rendered_text();
+ my $results = $html->get_results();
+
+ # end-of-document result values that require looking at the text
+
+ # count the number of spaces in the rendered text
+ my $space;
+ if (utf8::is_utf8($rendered)) {
+ my $str = $rendered;
+ $str =~ s/\S+//g; # delete non-whitespace Unicode characters
+ $space = length $str; # count remaining Unicode space characters
+ undef $str; # deallocate storage
+ dbg("message: spaces (Unicode) in HTML: %d out of %d%s",
+ $space, length $rendered,
+ $character_semantics ? '' : ', octets!?');
+ } else {
+ my $str = $rendered;
+ $space = $str =~ tr/ \t\n\r\x0b//;
+ dbg("message: spaces (octets) in HTML: %d out of %d%s",
+ $space, length $rendered,
+ $character_semantics ? ', chars!?' : '');
+ }
+ # we may want to add the count of other Unicode whitespace characters
+
+ $results->{html_length} = length $rendered; # perl characters count
+ $results->{non_space_len} = $results->{html_length} - $space;
+ $results->{ratio} = ($text_len - $results->{html_length}) / $text_len;
+
+ $node->{html_results} = $results;
+ # Pass the type explicitly: an HTML part renders as text/html whatever its
+ # declared type was, and Message::get_body_text_array_common keys the
+ # {metadata}{html} bookkeeping off that.
+ $node->set_rendered($rendered, 'text/html',
+ $html->get_rendered_text(invisible => 1),
+ $html->get_rendered_text(invisible => 0));
+}
+
1;
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Handler/SVG.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Handler/SVG.pm Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Handler/SVG.pm Tue Aug 25 21:16:50 2026 (r1937456)
@@ -164,6 +164,14 @@ sub handle_svg {
push @{ $pms->{Handler}{SVG}{text} }, @$text if @$text;
+ # Render the SVG's text into the node so ordinary body rules can match it.
+ # Node::rendered() renders text/plain only; every other type, this one
+ # included, gets its body text from whatever its handler publishes here.
+ # Use the part's own (image) type rather than the default text/plain, as
+ # Handler::Image does: get_body_text_array_common drops text/* parts that
+ # arrived as attachments, which is the common case for an SVG.
+ $node->set_rendered(join("\n", @$text), $node->effective_type) if @$text;
+
# Add links found in the SVG to the URI detail list, tagged 'svg' (so rules can
# target links that hide inside an "image") plus the source element name (e.g.
# 'a' for a clickable link, 'image' for an external image reference).
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Message.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Message.pm Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Message.pm Tue Aug 25 21:16:50 2026 (r1937456)
@@ -1309,24 +1309,9 @@ sub apply_handlers {
my $conf = $permsgstatus->{conf};
- # text/html parts only yield URIs (href/src/action/etc.) if a handler is
- # registered for them; without one (missing "loadhandler
- # Mail::SpamAssassin::Handler::HTML" in a .pre file), those URIs silently
- # never reach get_uri_detail_list(), hiding them from url_redirector*,
- # URIBL/URIDNSBL, and uri rules alike. Warn once so this doesn't go
- # unnoticed.
- if (!$warned_no_html_handler
- && !Mail::SpamAssassin::Conf::get_handler_for_type($conf, 'text/html')
- && $self->find_parts(qr/^text\/html$/, 1))
- {
- $warned_no_html_handler = 1;
- warn "message: no MIME handler registered for text/html -- URIs in HTML ".
- "links (href/src/action/etc.) will not be extracted; add ".
- "'loadhandler Mail::SpamAssassin::Handler::HTML' to a .pre file\n";
- }
-
- return unless $conf->{handlers} && %{$conf->{handlers}};
-
+ # Note we walk the parts even with an empty handler registry: the loop is a
+ # no-op then, but it is also where the missing-text/html-handler warning
+ # below fires, and "no handlers at all" is exactly a case worth warning about.
$self->parse_body() if exists $self->{'parse_queue'};
my $ctx = {
@@ -1349,9 +1334,26 @@ sub apply_handlers {
last if $ctx->{parts_budget} <= 0;
next if $depth > $ctx->{max_depth};
- my $handler =
- Mail::SpamAssassin::Conf::get_handler_for_type($conf, $node->effective_type);
- next unless $handler; # [ $plugin_obj, $methodname ]
+ my $type = $node->effective_type;
+ my $handler = Mail::SpamAssassin::Conf::get_handler_for_type($conf, $type);
+ if (!$handler) {
+ # A text/html part is parsed by its handler and by nothing else: with none
+ # registered (missing "loadhandler Mail::SpamAssassin::Handler::HTML" in a
+ # .pre file) the part is never rendered at all -- no body text for body
+ # rules or Bayes, no html_results for the html_* eval rules, and no URIs
+ # from href/src/action reaching get_uri_detail_list(), hiding them from
+ # url_redirector*, URIBL/URIDNSBL and uri rules alike. v403.pre loads the
+ # handler by default, so this normally only means an install whose lib/
+ # and rules/ are out of step. Warn once so it doesn't go unnoticed.
+ if ($type eq 'text/html' && !$warned_no_html_handler) {
+ $warned_no_html_handler = 1;
+ warn "message: no MIME handler registered for text/html -- HTML parts ".
+ "will not be rendered: no body text, no html_* rule results, and ".
+ "no URIs from HTML links (href/src/action/etc.); add ".
+ "'loadhandler Mail::SpamAssassin::Handler::HTML' to a .pre file\n";
+ }
+ next;
+ }
my $parts = $self->_invoke_handler($handler, $node, $ctx);
next unless $parts && @$parts;
@@ -1512,7 +1514,7 @@ sub get_body_text_array_common {
# right now, just use the last one. we may need to give some priority
# at some point, ie: use text/html rendered if it exists, or
# text/plain rendered as html otherwise.
- if ($html_needs_setting && $type eq 'text/html') {
+ if ($html_needs_setting && $type eq 'text/html' && $p->{html_results}) {
$self->{metadata}->{html} = $p->{html_results};
push @{$self->{metadata}->{html_all}}, $p->{html_results};
}
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Message/Node.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Message/Node.pm Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Message/Node.pm Tue Aug 25 21:16:50 2026 (r1937456)
@@ -448,6 +448,77 @@ sub decode {
}
}
+=item decode_and_normalize()
+
+Decode this part (as C<decode()> does) and, where the declared character set
+allows it, transcode the result from bytes to perl characters.
+
+Returns two scalars: the text, and a flag that is true when the text is in
+perl characters (utf8 flag on) and false when it is still bytes. Callers that
+do not care about the distinction can ignore the second value; those feeding
+the text to a parser that must be told which it is getting (e.g.
+L<Mail::SpamAssassin::HTML>) need it.
+
+Whether transcoding happens at all depends on the C<normalize_charset>
+setting and on the declared charset: with normalization off, only text that
+is declared US-ASCII or UTF-8 and really does decode as UTF-8 is converted.
+
+=cut
+
+sub decode_and_normalize {
+ my ($self) = @_;
+
+ my $text = $self->decode;
+ return (undef, 0) if !defined $text;
+
+ my $charset = $self->{charset};
+ if (!defined $charset) {
+ dbg("message: no charset declared, using us-ascii");
+ $charset = 'us-ascii';
+ }
+
+ # Is the returned text in perl characters, or still bytes?
+ my $character_semantics = 0; # $text is in bytes
+ if ($self->{normalize} && $enc_utf8) { # charset decoding requested
+ # Return Unicode characters rather than octets. For HTML this also avoids
+ # a HTML::Parser bug in utf8_mode
+ # https://rt.cpan.org/Public/Bug/Display.html?id=99755
+ # Note: the above bug was fixed in HTML-Parser 3.72, January 2016.
+ # Avoid unnecessary step of encoding-then-decoding by telling
+ # subroutine _normalize() to return Unicode text. See Bug 7133
+ #
+ $character_semantics = 1; # $text will be in characters
+ $text = _normalize($text, $charset, 1); # bytes to chars
+ } elsif ($charset =~ /^(?:US-ASCII|UTF-8)\z/i) {
+ if ($text !~ tr/\x00-\x7F//c) {
+ # all-ASCII, keep as octets (utf8 flag off)
+ dbg("message: contains only US-ASCII characters, declared %s, not decoding",
+ $charset);
+ } else { # non-ASCII, try UTF-8
+ my $rv;
+ # with some luck input can be interpreted as UTF-8
+ if (eval { $rv = $enc_utf8->decode($text, Encode::FB_CROAK | Encode::LEAVE_SRC); defined $rv }) {
+ $text = $rv; # decoded to perl characters
+ $character_semantics = 1; # $text will be in characters
+ dbg("message: decoded as charset UTF-8, declared %s", $charset);
+ } else {
+ my $err = '';
+ if ($@) {
+ $err = $@; $err =~ s/\s+/ /gs; $err =~ s/(.*) at .*/$1/;
+ $err = " ($err)";
+ }
+ dbg("message: failed decoding as charset UTF-8, declared %s%s",
+ $charset, $err);
+ }
+ }
+ } else {
+ dbg("message: 'normalize_charset' is off, encoding will likely ".
+ "be misinterpreted; declared charset: %s", $charset);
+ }
+
+ return ($text, $character_semantics);
+}
+
# Detect endianness of UTF-16 encoded data
sub detect_utf16 {
my $utf16le_clues = 0;
@@ -788,169 +859,77 @@ sub effective_type {
=item rendered()
-rendered() takes the given text/* type MIME part, and attempts to
-render it into a text scalar. It will always render text/html, and will
-use a heuristic to determine if other text/* parts should be considered
-text/html. Two scalars are returned: the rendered type (either text/html
-or whatever the original type was), and the rendered text.
+Returns the rendered text for this part, as two scalars: the rendered type
+and the rendered text.
+
+text/plain parts are rendered here (charset-decoded, see
+C<decode_and_normalize()>). Every other type is rendered by its registered
+MIME-part handler, which publishes the result with C<set_rendered()> during
+C<apply_handlers()>; this method then just returns what the handler cached.
+text/html, for instance, is rendered by
+L<Mail::SpamAssassin::Handler::HTML>.
+
+Returns C<(undef,undef)> for a part that is neither text/plain nor rendered by
+a handler -- including a text/html part reached before the handlers have run,
+or when no handler is registered for its type.
=cut
sub rendered {
my ($self) = @_;
- # Cached?
+ # Cached? Either rendered below on an earlier call, or published by the
+ # part's MIME handler via set_rendered().
if (exists $self->{rendered}) {
return ($self->{rendered_type}, $self->{rendered});
}
- # We only know how to render text/plain and text/html ...
+ # We only render text/plain here; everything else is a handler's job.
# Note: for bug 4843, make sure to skip text/calendar parts
# we also want to skip things like text/x-vcard
# text/x-aol is ignored here, but looks like text/html ...
my $type = $self->effective_type();
- unless ($type eq 'text/plain' || $type eq 'text/html' || $type eq 'image/svg+xml') {
- return (undef,undef);
- }
-
- my $text = $self->decode; # QP and Base64 decoding, bytes
- my $text_len = length($text); # num of bytes in original charset encoding
-
- my $charset = $self->{charset};
- if (!defined $charset) {
- dbg("message: no charset declared, using us-ascii");
- $charset = 'us-ascii';
- }
-
- # render text/html always
- if ($text ne '' && $type ne 'text/plain' )
- {
- $self->{rendered_type} = 'text/html';
-
- # will input text to HTML::Parser be provided as Unicode characters?
- my $character_semantics = 0; # $text is in bytes
- if ($self->{normalize} && $enc_utf8) { # charset decoding requested
- # Provide input to HTML::Parser as Unicode characters
- # which avoids a HTML::Parser bug in utf8_mode
- # https://rt.cpan.org/Public/Bug/Display.html?id=99755
- # Note: the above bug was fixed in HTML-Parser 3.72, January 2016.
- # Avoid unnecessary step of encoding-then-decoding by telling
- # subroutine _normalize() to return Unicode text. See Bug 7133
- #
- $character_semantics = 1; # $text will be in characters
- $text = _normalize($text, $charset, 1); # bytes to chars
- } elsif ($charset =~ /^(?:US-ASCII|UTF-8)\z/i) {
- if ($text !~ tr/\x00-\x7F//c) {
- # all-ASCII, keep as octets (utf8 flag off)
- dbg("message: contains only US-ASCII characters, declared %s, not decoding",
- $charset);
- } else { # non-ASCII, try UTF-8
- my $rv;
- # with some luck input can be interpreted as UTF-8
- if (eval { $rv = $enc_utf8->decode($text, Encode::FB_CROAK | Encode::LEAVE_SRC); defined $rv }) {
- $text = $rv; # decoded to perl characters
- $character_semantics = 1; # $text will be in characters
- dbg("message: decoded as charset UTF-8, declared %s", $charset);
- } else {
- my $err = '';
- if ($@) {
- $err = $@; $err =~ s/\s+/ /gs; $err =~ s/(.*) at .*/$1/;
- $err = " ($err)";
- }
- dbg("message: failed decoding as charset UTF-8, declared %s%s",
- $charset, $err);
- }
- }
- } else {
- dbg("message: 'normalize_charset' is off, encoding will likely ".
- "be misinterpreted; declared charset: %s", $charset);
- }
- # the 1 requires decoded HTML results to be in characters (utf8 flag on)
- my $html = Mail::SpamAssassin::HTML->new($character_semantics,1); # object
+ return (undef,undef) if $type ne 'text/plain';
- $html->parse($text); # parse+render text
+ my ($text) = $self->decode_and_normalize();
+ return (undef,undef) if !defined $text;
- # resulting HTML-decoded text is in perl characters (utf8 flag on)
- $self->{rendered} = $html->get_rendered_text();
- $self->{visible_rendered} = $html->get_rendered_text(invisible => 0);
- $self->{invisible_rendered} = $html->get_rendered_text(invisible => 1);
- $self->{html_results} = $html->get_results();
-
- # end-of-document result values that require looking at the text
- my $r = $self->{html_results}; # temporary reference for brevity
-
- # count the number of spaces in the rendered text
- my $space;
- if (utf8::is_utf8($self->{rendered})) {
- my $str = $self->{rendered};
- $str =~ s/\S+//g; # delete non-whitespace Unicode characters
- $space = length $str; # count remaining Unicode space characters
- undef $str; # deallocate storage
- dbg("message: spaces (Unicode) in HTML: %d out of %d%s",
- $space, length $self->{rendered},
- $character_semantics ? '' : ', octets!?');
- } else {
- my $str = $self->{rendered};
- $space = $str =~ tr/ \t\n\r\x0b//;
- dbg("message: spaces (octets) in HTML: %d out of %d%s",
- $space, length $self->{rendered},
- $character_semantics ? ', chars!?' : '');
- }
- # we may want to add the count of other Unicode whitespace characters
-
- $r->{html_length} = length $self->{rendered}; # perl characters count
- $r->{non_space_len} = $r->{html_length} - $space;
- $r->{ratio} = ($text_len - $r->{html_length}) / $text_len if $text_len;
- }
- else { # plain text
- if ($self->{normalize} && $enc_utf8) {
- # request transcoded result as UTF-8 octets!
- $text = _normalize($text, $charset, 1); # bytes to chars
- } elsif ($charset =~ /^(?:US-ASCII|UTF-8)\z/i) {
- if ($text =~ tr/\x00-\x7F//c) { # non-ASCII, try UTF-8
- my $rv;
- # with some luck input can be interpreted as UTF-8
- if (eval { $rv = $enc_utf8->decode($text, Encode::FB_CROAK | Encode::LEAVE_SRC); defined $rv }) {
- $text = $rv; # decoded to perl characters
- dbg("message: decoded as charset UTF-8, declared %s", $charset);
- } else {
- my $err = '';
- if ($@) {
- $err = $@; $err =~ s/\s+/ /gs; $err =~ s/(.*) at .*/$1/;
- $err = " ($err)";
- }
- dbg("message: failed decoding as charset UTF-8, declared %s%s",
- $charset, $err);
- }
- } else {
- dbg("message: contains only US-ASCII characters, declared %s, not decoding",
- $charset);
- }
- }
- $self->{rendered_type} = $type;
- $self->{rendered} = $self->{visible_rendered} = $text;
- $self->{invisible_rendered} = '';
- }
+ $self->{rendered_type} = $type;
+ $self->{rendered} = $self->{visible_rendered} = $text;
+ $self->{invisible_rendered} = '';
return ($self->{rendered_type}, $self->{rendered});
}
-=item set_rendered($text, $type)
+=item set_rendered($text, $type, $invisible, $visible)
Set the rendered text and type for the given part. If type is not
specified, and text is a defined value, a default of 'text/plain' is used.
-This can be used, for instance, to render non-text parts using plugins.
+This is how a MIME-part handler publishes the text it extracted, so that
+ordinary body rules can match it.
+
+C<$invisible> and C<$visible> are optional. A caller whose renderer
+distinguishes text the recipient can see from text it cannot (currently
+L<Mail::SpamAssassin::Handler::HTML>) passes both, where C<$text> is
+everything and C<$visible> is the visible subset. Omit them and all of
+C<$text> counts as visible, with no invisible text -- the right answer for a
+renderer with no notion of hidden content, such as OCR or document text
+extraction.
=cut
sub set_rendered {
- my ($self, $text, $type) = @_;
+ my ($self, $text, $type, $invisible, $visible) = @_;
$type = 'text/plain' if (!defined $type && defined $text);
$self->{'rendered_type'} = $type;
- $self->{'rendered'} = $self->{'visible_rendered'} = $text;
- $self->{'invisible_rendered'} = defined $text ? '' : undef;
+ $self->{'rendered'} = $text;
+ # Only a renderer that can tell hidden text from shown text passes these;
+ # for everyone else all of the rendered text is visible.
+ $self->{'visible_rendered'} = defined $visible ? $visible : $text;
+ $self->{'invisible_rendered'} =
+ defined $invisible ? $invisible : (defined $text ? '' : undef);
}
=item visible_rendered()
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Bayes.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Bayes.pm Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/Bayes.pm Tue Aug 25 21:16:50 2026 (r1937456)
@@ -1100,6 +1100,17 @@ sub get_body_from_msg {
my $permsgstatus =
Mail::SpamAssassin::PerMsgStatus->new($self->{main}, $msg);
$msg->extract_message_metadata ($permsgstatus);
+ # Run the MIME-part handlers, so that the text they extract (PDF, OCR'd
+ # images, calendar invites) is part of what Bayes learns, the same text body
+ # rules match against. _get_msgdata_from_permsgstatus() below already brings
+ # these about by way of get_uri_list(), but only as a side effect of wanting
+ # URIs; ask for them directly so the token stream does not quietly depend on
+ # that. Idempotent, so the later call is a no-op.
+ # (Note extract_message_metadata() above is Message::, which does not run
+ # handlers; the PerMsgStatus:: method of the same name does, but also fires
+ # the parsed_metadata plugin callbacks -- DNS lookups the learn path has no
+ # use for.)
+ $msg->apply_handlers($permsgstatus);
my $msgdata = $self->_get_msgdata_from_permsgstatus ($permsgstatus);
$permsgstatus->finish();
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Pyzor/Digest/Pieces.pm Tue Aug 25 21:16:50 2026 (r1937456)
@@ -97,7 +97,10 @@ sub digest_payloads {
$payload = $p->{'decoded'};
$payload =~ s/\\'/\'/gx;
} else {
- $payload = $p->{'rendered'};
+ # rendered() rather than a raw {rendered} read: for a text/html
+ # part the text is put there by the HTML handler, and reading the
+ # key directly only works if that has already happened.
+ (undef, $payload) = $p->rendered();
}
utf8::upgrade($payload) if defined $payload;
Modified: spamassassin/trunk/t/scan_text_attach.t
==============================================================================
--- spamassassin/trunk/t/scan_text_attach.t Tue Aug 25 20:27:52 2026 (r1937455)
+++ spamassassin/trunk/t/scan_text_attach.t Tue Aug 25 21:16:50 2026 (r1937456)
@@ -6,6 +6,7 @@ use SATest; sa_t_init("scan_text_attach"
use Test::More tests => 8;
use Mail::SpamAssassin;
+use Mail::SpamAssassin::PerMsgStatus;
# A text/html part marked Content-Disposition: attachment is, by default,
# excluded from the rendered body text that body rules see. The
@@ -30,10 +31,18 @@ sub bodies {
open (IN, "<data/nice/scan_text_attach") or die "cannot open fixture: $!";
my $mail = $sa->parse(\*IN);
close IN;
+ # Run the MIME-part handlers, as a real scan does: a text/html part is
+ # rendered by Mail::SpamAssassin::Handler::HTML, so without this there is no
+ # rendered text for it at all. apply_handlers() needs a PerMsgStatus for the
+ # handlers to accumulate findings on, even though this test reads only the
+ # body arrays off the message.
+ my $pms = Mail::SpamAssassin::PerMsgStatus->new($sa, $mail);
+ $mail->apply_handlers($pms);
my %b = (
rendered => join('||', @{$mail->get_rendered_body_text_array()}),
invisible => join('||', @{$mail->get_invisible_rendered_body_text_array()}),
);
+ $pms->finish;
$mail->finish;
$sa->finish;
return %b;
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.