Author: fkento
Date: Mon Aug 17 05:00:51 2026
New Revision: 1937171
Log:
PDF: extract CCITT fax images by wrapping them in a TIFF header
Faxed PDFs encode their page images with CCITTFaxDecode (Group 3/4 fax),
as do monochrome scans, and _get_image_data had no branch for it: it fell
through to 'unsupported', extract_images dropped the image, and no
sub-part ever reached the image handler. A PDF whose pages are all fax
images was therefore invisible to OCR.
Rather than implementing T.4/T.6, leave the data encoded and wrap it in a
TIFF header, which image readers decode natively (libtiff, via leptonica,
in tesseract). The codec parameters live in /DecodeParms rather than the
stream, so the wrapping happens in the parser while they are in hand; the
descriptor then carries a complete image file, as 'jpeg' already did, and
the handler passes it through as image/tiff.
/K maps onto the TIFF compression tag (Group 4, Group 3 1-D, Group 3 2-D)
and BlackIs1 onto PhotometricInterpretation. Group 4 combined with
EncodedByteAlign has no TIFF equivalent and is declined rather than
wrapped into data that would decode to garbage.
Also add a TIFF branch to Handler::Image::_image_info. Without it a TIFF
sub-part is an unrecognised type, and the type gate skips OCR entirely.
This also means genuine image/tiff mail attachments now OCR, where before
they were silently dropped.
Added:
spamassassin/trunk/t/data/pdf/ccitt_g4.pdf (contents, props changed)
Modified:
spamassassin/trunk/MANIFEST
spamassassin/trunk/lib/Mail/SpamAssassin/Handler/Image.pm
spamassassin/trunk/lib/Mail/SpamAssassin/Handler/PDF.pm
spamassassin/trunk/lib/Mail/SpamAssassin/PDF/Parser.pm
spamassassin/trunk/rules/v402.pre
Modified: spamassassin/trunk/MANIFEST
==============================================================================
--- spamassassin/trunk/MANIFEST Mon Aug 17 04:20:06 2026 (r1937170)
+++ spamassassin/trunk/MANIFEST Mon Aug 17 05:00:51 2026 (r1937171)
@@ -474,6 +474,7 @@ t/data/pdf/aesv2.pdf
t/data/pdf/aesv2_cleartext_meta.pdf
t/data/pdf/aesv3.pdf
t/data/pdf/ascii85.pdf
+t/data/pdf/ccitt_g4.pdf
t/data/pdf/indirect_mediabox.pdf
t/data/pdf/inline_image.pdf
t/data/pdf/link_page2.pdf
@@ -680,6 +681,7 @@ t/olevbmacro.t
t/originating_ip_hdr.t
t/parameter_header.t
t/pdf_assert_token.t
+t/pdf_ccitt_tiff.t
t/pdf_extract_images.t
t/pdf_get_array.t
t/pdf_get_dict.t
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Handler/Image.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Handler/Image.pm Mon Aug 17 04:20:06 2026 (r1937170)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Handler/Image.pm Mon Aug 17 05:00:51 2026 (r1937171)
@@ -489,6 +489,38 @@ sub _image_info {
return ('bmp', ($w && $h) ? ($w, $h) : (undef, undef));
}
+ # TIFF: "II\x2a\x00" (little-endian) or "MM\x00\x2a" (big-endian), then the
+ # offset of the first IFD. Dimensions come from the ImageWidth (256) and
+ # ImageLength (257) tags, which may be SHORT or LONG. Only the first IFD is
+ # read: a multi-page TIFF is measured by its first page, which is what an image
+ # reader shows and enough for the size gate.
+ if (substr($$dataref, 0, 4) eq "II\x2a\x00" ||
+ substr($$dataref, 0, 4) eq "MM\x00\x2a") {
+ my $le = substr($$dataref, 0, 2) eq 'II';
+ my $ifd = unpack($le ? 'V' : 'N', substr($$dataref, 4, 4));
+ my ($w, $h);
+ # Guard every read against a truncated or bogus offset; a malformed TIFF must
+ # fall through as a known type with unknown dimensions, not die.
+ if ($ifd >= 8 && $ifd + 2 <= $len) {
+ my $count = unpack($le ? 'v' : 'n', substr($$dataref, $ifd, 2));
+ $count = 512 if $count > 512; # sanity cap on entries to walk
+ for my $i (0 .. $count - 1) {
+ my $e = $ifd + 2 + $i * 12;
+ last if $e + 12 > $len;
+ my ($tag, $type) = unpack($le ? 'vv' : 'nn', substr($$dataref, $e, 4));
+ next unless $tag == 256 || $tag == 257;
+ # SHORT (3) sits in the low half of the value field; LONG (4) fills it.
+ my $v = $type == 3 ? unpack($le ? 'v' : 'n', substr($$dataref, $e + 8, 2))
+ : $type == 4 ? unpack($le ? 'V' : 'N', substr($$dataref, $e + 8, 4))
+ : undef;
+ next unless defined $v;
+ $tag == 256 ? ($w = $v) : ($h = $v);
+ last if defined $w && defined $h;
+ }
+ }
+ return ('tiff', ($w && $h) ? ($w, $h) : (undef, undef));
+ }
+
# ISO-BMFF / HEIF: bytes 4-7 are 'ftyp', bytes 8-11 a brand code. Identified
# for conversion; dimensions read from the converted PNG, not here.
if (substr($$dataref, 4, 4) eq 'ftyp') {
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/Handler/PDF.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/Handler/PDF.pm Mon Aug 17 04:20:06 2026 (r1937170)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/Handler/PDF.pm Mon Aug 17 05:00:51 2026 (r1937171)
@@ -39,9 +39,10 @@ PDFInfo plugin.
The handler returns embedded images extracted from the PDF as sub-parts, each a
C<< { type => '<mediatype>', data => $bytes } >> spec that the handler framework
-dispatches to the image handler (e.g. for OCR). Images are emitted as either
-C<image/jpeg> (for streams already in JPEG form) or C<image/png> (raw image data
-re-wrapped as PNG).
+dispatches to the image handler (e.g. for OCR). Images are emitted as
+C<image/jpeg> (for streams already in JPEG form), C<image/tiff> (for CCITT fax
+data, which is left encoded and wrapped in a TIFF header the image reader decodes
+natively), or C<image/png> (raw image data re-wrapped as PNG).
This extraction is gated by the C<pdf_extract_images>, C<pdf_max_images>, and
C<pdf_max_image_pixels> settings (see L</CONFIGURATION>), and is skipped entirely
@@ -536,6 +537,8 @@ sub _extract_text {
# image can't be represented in a format the image handler reads.
#
# format 'jpeg' - already a complete JPEG file, passed through as image/jpeg
+# format 'tiff' - already a complete TIFF file (CCITT fax data wrapped by the
+# parser), passed through as image/tiff
# format 'raw' - raw samples, wrapped into a PNG (image/png). Only DeviceGray,
# DeviceRGB (8bpc) and bilevel (1bpc) are handled; CMYK/Indexed/
# array colorspaces are skipped.
@@ -544,10 +547,10 @@ sub _encode_image {
my $bytes = $img->{bytes};
return undef unless defined $bytes && length $bytes;
- if ( ($img->{format} // '') eq 'jpeg' ) {
- return { type => 'image/jpeg', data => $bytes };
- }
- return undef unless ($img->{format} // '') eq 'raw';
+ my $format = $img->{format} // '';
+ return { type => 'image/jpeg', data => $bytes } if $format eq 'jpeg';
+ return { type => 'image/tiff', data => $bytes } if $format eq 'tiff';
+ return undef unless $format eq 'raw';
my $w = $img->{width};
my $h = $img->{height};
Modified: spamassassin/trunk/lib/Mail/SpamAssassin/PDF/Parser.pm
==============================================================================
--- spamassassin/trunk/lib/Mail/SpamAssassin/PDF/Parser.pm Mon Aug 17 04:20:06 2026 (r1937170)
+++ spamassassin/trunk/lib/Mail/SpamAssassin/PDF/Parser.pm Mon Aug 17 05:00:51 2026 (r1937171)
@@ -782,8 +782,10 @@ sub _get_stream_data {
# 'raw' - fully decoded raw samples (Flate/LZW/ASCII85 chain completed); $bytes
# are the raw samples, to be wrapped in an image header by the caller
# 'jpeg' - DCTDecode reached; $bytes are a complete JPEG file, usable as-is
-# 'unsupported' - reached a codec we can't undo and can't pass through (CCITTFax/JPX/
-# RunLength/etc.); $bytes is undef because the data so far is useless
+# 'tiff' - CCITTFaxDecode reached; $bytes are a complete TIFF file wrapping the
+# still-encoded fax data, usable as-is
+# 'unsupported' - reached a codec we can't undo and can't pass through (JPX/RunLength/
+# etc.); $bytes is undef because the data so far is useless
# (neither raw samples nor a standalone image file)
# Returns () if the object isn't a readable stream.
#
@@ -833,8 +835,29 @@ sub _get_image_data {
} elsif ( $filter eq '/DCTDecode' ) {
# Whatever we have so far is a complete JPEG file; return it as-is.
return ('jpeg', $stream_data);
+ } elsif ( $filter eq '/CCITTFaxDecode' ) {
+ # Group 3/4 fax, the usual encoding for scanned documents. Rather than
+ # decoding T.4/T.6 here, the still-encoded data is wrapped in a TIFF
+ # header, which image readers decode natively. The codec parameters are
+ # not in the stream but in /DecodeParms, so the wrapping happens here
+ # while they are in hand; Columns/Rows fall back to the image dictionary.
+ $decodeParms = {} unless ref($decodeParms) eq 'HASH';
+ my %parms = (
+ K => $self->_dereference($decodeParms->{'/K'}) // 0,
+ Columns => $self->_dereference($decodeParms->{'/Columns'}) // 1728,
+ Rows => $self->_dereference($decodeParms->{'/Rows'}),
+ BlackIs1 => $self->_dereference($decodeParms->{'/BlackIs1'}) // 0,
+ EncodedByteAlign => $self->_dereference($decodeParms->{'/EncodedByteAlign'}) // 0,
+ );
+ $parms{Rows} = $self->_dereference($stream_obj->{'/Height'})
+ unless defined $parms{Rows};
+ my $tiff = _ccitt_to_tiff($stream_data, \%parms);
+ # Parameters TIFF can't express (see _ccitt_to_tiff); wrapping anyway
+ # would decode to garbage.
+ return ('unsupported', undef) unless defined $tiff;
+ return ('tiff', $tiff);
} else {
- # CCITTFaxDecode, JPXDecode, RunLengthDecode, etc. - can't decode (yet)
+ # JPXDecode, RunLengthDecode, etc. - can't decode (yet)
return ('unsupported', undef);
}
}
@@ -842,10 +865,87 @@ sub _get_image_data {
return ('raw', $stream_data);
}
+# _ccitt_to_tiff($bytes, $parms): wrap still-encoded CCITT fax data in a TIFF header.
+# The fax codecs carry no dimensions or polarity of their own -- those live in the
+# PDF's /DecodeParms -- so the bytes are useless to an image reader until described
+# by a container. Emitting TIFF lets the reader's own fax decoder (libtiff, via
+# leptonica in tesseract) do the work rather than reimplementing T.4/T.6 here.
+#
+# Maps the PDF /K parameter onto the TIFF compression tag:
+# K < 0 Group 4 / T.6 -> Compression 4
+# K == 0 Group 3 1-D -> Compression 3, T4Options bit0 clear
+# K > 0 Group 3 mixed 1-D/2-D -> Compression 3, T4Options bit0 set
+#
+# Returns the TIFF bytes, or undef if the parameters can't be expressed in TIFF.
+sub _ccitt_to_tiff {
+ my ($bytes, $parms) = @_;
+ $parms ||= {};
+
+ my $w = $parms->{Columns};
+ my $h = $parms->{Rows};
+ return undef unless defined($w) && defined($h) && $w > 0 && $h > 0;
+
+ my $k = $parms->{K} || 0;
+ my ($compression, $t4options);
+ if ( $k < 0 ) { $compression = 4 }
+ elsif ( $k == 0 ) { $compression = 3; $t4options = 0 }
+ else { $compression = 3; $t4options = 1 }
+
+ # EncodedByteAlign has no T.6 equivalent; libtiff only honours it for Group 3
+ # (T4Options bit 2). A Group 4 stream that sets it can't be described in TIFF,
+ # so decline rather than hand over data that would decode to garbage.
+ if ( $parms->{EncodedByteAlign} ) {
+ return undef if $compression == 4;
+ $t4options |= 0x4;
+ }
+
+ # PDF BlackIs1 false (the default) means 0 = black, i.e. TIFF's WhiteIsZero.
+ my $photometric = $parms->{BlackIs1} ? 1 : 0; # 1 = MinIsBlack, 0 = MinIsWhite
+
+ my @tags = (
+ [ 256, 4, $w ], # ImageWidth
+ [ 257, 4, $h ], # ImageLength
+ [ 258, 3, 1 ], # BitsPerSample
+ [ 259, 3, $compression ], # Compression
+ [ 262, 3, $photometric ], # PhotometricInterpretation
+ [ 273, 4, 0 ], # StripOffsets (patched below)
+ [ 277, 3, 1 ], # SamplesPerPixel
+ [ 278, 4, $h ], # RowsPerStrip (single strip)
+ [ 279, 4, length $bytes ], # StripByteCounts
+ );
+ push @tags, [ 292, 4, $t4options ] if defined $t4options; # T4Options
+ @tags = sort { $a->[0] <=> $b->[0] } @tags; # TIFF requires ascending tags
+
+ # Little-endian TIFF: header (8) + entry count (2) + 12 per entry + next-IFD (4).
+ # All values here are scalars that fit in the 4-byte inline field, so there is
+ # no overflow area and the image data follows the IFD directly.
+ my $ifd_offset = 8;
+ my $data_offset = $ifd_offset + 2 + 12 * scalar(@tags) + 4;
+ for my $t ( @tags ) {
+ $t->[2] = $data_offset if $t->[0] == 273; # patch StripOffsets
+ }
+
+ my $tiff = "II\x2a\x00" . pack('V', $ifd_offset);
+ $tiff .= pack('v', scalar @tags);
+ for my $t ( @tags ) {
+ my ($tag, $type, $value) = @$t;
+ # Values shorter than 4 bytes sit left-justified in the value field; packing
+ # SHORT as 'vv' places it in the low half with the high half zeroed.
+ $tiff .= pack('vvV', $tag, $type, 1)
+ . ($type == 3 ? pack('vv', $value, 0) : pack('V', $value));
+ }
+ $tiff .= pack('V', 0); # no next IFD
+ $tiff .= $bytes;
+
+ return $tiff;
+}
+
#
# Decode the XObject images collected during parsing into descriptors ready for the
# caller to re-encode (e.g. as PNG/JPEG sub-parts). Returns an arrayref of hashrefs:
-# { format => 'raw'|'jpeg', bytes => $data, width, height, colorspace, bpc }
+# { format => 'raw'|'jpeg'|'tiff', bytes => $data, width, height, colorspace, bpc }
+# 'jpeg' and 'tiff' bytes are complete image files, ready to use as-is; 'raw' bytes
+# are samples the caller wraps in an image header.
# Honors caps:
# max_images - stop after this many usable images (default 4)
# max_pixels - skip images larger than this (width*height) (default 25_000_000;
Modified: spamassassin/trunk/rules/v402.pre
==============================================================================
--- spamassassin/trunk/rules/v402.pre Mon Aug 17 04:20:06 2026 (r1937170)
+++ spamassassin/trunk/rules/v402.pre Mon Aug 17 05:00:51 2026 (r1937171)
@@ -21,4 +21,4 @@
# Note that this plugin will send HTTP requests to different URL redirector
# services. Enabling caching is recommended, see plugin documentation.
#
-# loadplugin Mail::SpamAssassin::Plugin::Redirectors
+loadplugin Mail::SpamAssassin::Plugin::Redirectors
Added: spamassassin/trunk/t/data/pdf/ccitt_g4.pdf
==============================================================================
Binary file. No diff available.
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.